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ABSTRACT 

This publication describes a practicum project that 
developed a comprehensive educational evaluation system for 
coi'lectinge storing^ ^nd displaying pertinent data for use in 
planning educational programs at both the di^^rict and school level 
in the City School District' of New fiochelle> The resulting New 
P.oche.lle Evaluation Mo^el was developed from Stuff lebeam* s CIPP 
models incorporating aspects of Provus* Discrepancy model and data 
collection methods emphasizefi by Stake. The model has been used on a 
districtwide level to evaluate ail programs for students with special 
needs and has also been used to evaluate the reading programs in two 
elementary schools. Sample data collection forms and a users^ guide 
to tfce New Rochelle Evaluation Model are included in the appendix. 
(Author/JG) 
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• - ABSTRACT 

The City Sclfool District of New Rdchelle, the Pennington Blement^'ry 
School, Mount Vernon and the Hillcrest Elementary School, Spring Valley 
shared. a common need for more effective evaluation. The practicum parti- 
cipants, all employed in the districts mentioned, set out to fill this need 
by developing and implementing a workable evaluation model. 

The evaluation model adopted by the practicum participants was 
developed out of Stufflebeam's CIPP model and incorporated aspects of 
Provus' Discrepancy mode) and data collection methods emphasized by 
Stake . 

The resulting New Rochelle Evaluation Model has been implemented 
^^^^^ a dismct-wide level in the City School District of l^ew Rochelle, and 
on ajDuilding level in the Hillcrest Elementary School, Spring Valley, and 
the Pennington Elementary School, Mount Vernon. The district -wide. evalua- 
tion focused on all programs for students with special needs, while the 

r. 

building level evaluation focused on reading programs. 

The participants believe that the model they have developed will 
assist in their school's and district's making more effective educational 
decisions. 
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CHAPTER I 

INTRODUCTION: 
AN ANALYSIS OF THE SUBJECT DISTRICT'S CHARACTERISTICS 

The City of New Rochelle,* located in lower Westchester County, New 
York, is a suburb of 75,385 residents seventeen miles from the heart of New York 
City. The socio-econ> »mic status of the population ranges from the high and 
upper middle class to an ever-increasing number of low income families, many 
of whom are the working poor or are welfare recipiej^ts. According to most recent 
Title I figures, there are 1,214 AFDC children living in the city, 198 foster; 
children, and four of the city's ten elementary schools have a poverty-eligible 
enrollment of 30^. > 

.The community is a microcosm of the greater New York me^opolitan ar^a. 
the public schools' 11,681 students reflect the ethnic and socio-economic i 
diversity of the city. There is a sizable black minprity (10,854), a large Italian 
community, and a growing number of students entering the school system to wljiom 
English is a second language. (The primary language of this sub-group is usually 
Italian but there is a developing Spanish speaking minority also.) 

The New Rochelle School District is coterminous with the City of New' 
Rochelle and consists of one comprehensive high school, two Junior high schools, 
ten elementary schools and one learning center. Fourteen private and parochial 
schools enroll approximately 3,200 students. 

♦ 

A Federal Court decree in 1961 ordered the New Rochelle Board of Educa- 
tion to develop a desegregation plan for the predominantly black Lincoln Ele- 
mentary School. The Board* s plan allowed parents 0^ the Lincoln students to 

8 

L 
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transfer their qhlldren to other schools In the city, with the stipulations that. the 
transfer of children would not cause class sizes to exceed the existing class 
size averages in the rest of the city's schools, and that the ethnic balance in the 
receiving school would not exceed 50% black. In September 1961 approximately 
half the Uncoln student population was transferred to other elementary schools. 
During 1962, the Lincoln School remained open but with only half its original , 
.student body. By 1963 the District closed the school entirely with all the re- 
maining students having been transferred to Nevy Rochelle's then eleven other 
^ elementary scho6ls. At this same tim'e, the Board then provided transportation, 
for all the Lincoln students living from 1-1/2 to 10 miles from their new school. 
Previously Lincoln parents had contracted" directly for their own transportation. 

Today New Rochelle remains in compliance with the court or^er, with 
the elementary schools' black population ranging from 14 to 49 percent. Stu- 
dents from the former Lincoln School area continue to Be bussed in^o schools 
which serve basically white residential areas. But, while the school system 
is "desegregated" in terms of court order under which it operates, this is ac- 
complished by bussing children of whom the overwhelming majority are black 
• and from low income families into schools whose population is overwhelmingly 
white and well-to-do middle class. Because of inadequate resources and the 
shortcomings of traditional evaluation techniques, the district has been unable 
• effectively to evaluate tha impact of these forces on the academic achievement, 
emotional growth and racial attitudes of its students. Nor has the district been 
able to determine the effect of teacher and communit/ attitudes on the achieve- 
ment of its students. 
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, Events surrounding the closing of Lincoln School and movements in edu- 
cation on a local, state and^ national level served to focus attention on the -fact 
that the schools had^not^been successful in providing many of their students with 
basic and essential skills. In spite of the best efforts of a dedicated and know- 
ledgeable staff and the expenditure of considerable state and federal funds on 
special projects throughout these last years, it cannot be said^that the district's 
basic goal of "having every child acquire those basic skills which enable him t 
make the further educational choices which will best fulfill his desires and his 
abilities "'has been achieved. At this time tests ^how that 24% of the school 
population is performing below^ minimum competency levels in reading and 35% 

below that level in math. There is considerable parental dissatisfaction and 
' 1 

•disillusionment with present school performance, evidenced in part by the con- 
siderable number ^f children who attend p?^ate or parochial schools. Many of 
those parents who do fcive a choice of schools opt for the non-public schools. 
Within the public schools there seems to be a range in, the results pro* 
duced by exposure to New Rochelle's educational program. In the spring of 1971, 
"The New Rochelle School Pmfile: A Report of the New Rochelle Public Schools 
to the Community" was published^ (Appendix A). Perhaps the most salient fact 
about the elementary schools that came to public attention as a result of this 
study was the range of difference shown from one school to another when one 

- ( 

looks at any of the recorded achievement scores. One school recorded five per- 
cent of its 6th grade "below minimum competency" in reading; another ^school 
fifty percent. These problems, common to almost ail urban systems, persist 
even though the district now draws funds from many state and federal agencies 
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including ESEA Title I, Title II, Title III, Title IV, Aid to Urban Education, •ESAA 
Manpower Training, Educational Professional Development Act, Pre-Kindergarten, 
and NDEA III. Last year well over $1 ,000,000 came into the district ir categori- 
cal funding from these and other sources . 

In addition, the dist'lct has developed a variety of curricula and instruc- 
tional InnoVatlons including an extensive district-wide open education program 
at the elementary level, Mid-Start urograms for junior high schools, a Mini- 
Course program at the high school as well as one modeled after the Philadelphia 
Parkway Program. Also changes have occurred at the classroom level with the 
movement of a ijarge number of teachers towards individualization of instruction, 
personalized learning, differentiated staffing and teaming, IPI, Modular Sched- 
uling, etc. But again, at least in part because of a lack of effective evaluation 
procedures, little is certain about the effect 6f these programs on each student, 
his achievement or affective development. And, too, the administrative decision- 
making effectiveness is severely hampered by a lack of pertinent information on 
students, programs and curricula. ^ 

New Rochelle is not substantially different from thousands of other city 
and suburban public school systems in that the rising cost of operation has placed 
an unbearable burden on its local property taxpayers^ But it Is.dtfferent.in one__ 
potentially destructive v;oy: while most of the others have the latitude, however 
burdensome ^ to increase local revenues by raising taxes. New Rochelle is blocked 
from that option by the Constitution of the State of New York. It has reached the 
2.0 percent tax limit imposed by Article VIII, Sectiop-lO (c) on any school district 
which is coterminous with, partly within or wholly within a city having less than 
125,000 inhabitants. 



In the past, to reach the education objectives of the community, th^ 
people have voted to increase- their contributipn, through taxes, to the operation 
of the schools, and in recent years have authorized increases in the tax limit 
from 1.25 to 2.0 percent. Uoyi^ bepause of the Constitutional impediment Just 
mentioned, that way to better or morfe varied programs is closed to them. , ^ 
Ironically, although the community has demonstrated its ppwerful support for 
excellence in its schools over the years, the flexibility to determine its futiire, 
academic destiny has been seriously diminished. Forced to deliver mandated 
SCI vices and to meet State-ordered requirements in staff negotiations, course 
offerings, ^etc. , the district has exhausted its available locaJ resources. These 
financial constraints have Imposed upon New Rochelle the need to rethink and 
redesign many aspects of its school program. 

In 1971, for instance, the district wais faced with a budgetary deficit of 
$1.2 million dollars. The alternatives we'^e limited. The district could have 
adopted the usual standard measures ot cutting costs by incr^^^ing class size, 
cutting a.l! specialists, dropping or trimming programs, cutting service, supplies, 
etc. Instead, the district decided to use its fiscal problem as an opportunity to 
redesign education in thte district with two clear criteria in mind: 

1. to stay within budget limitation; ^ 

2. to provide a better program for children than had been provided 
previously by refining instructional delivery systems. 

Of course, the New Rochelle redesign had'to involve parents, teachers 
and community. The citizens, of New Rochelle had to see the impending cutbacks 
as providing a starting point, a base for improvement rather than- an attack on the 



schools. To effect this change the superintendent obtained a small grant from 
the Rockefeller Foundation to plan and carry o^t citizen participation in the 
redesign process by holding a series of conferences in the schools where teachers, 
parents, students and citizens could identify for themselves the problems faced 

by the district and the' objectives of the district's educational system. Fifty- 

2 . 
eight problem areas were identified by the group. Then the committee identi- 

^ 3 

fied 23 objectives. ' They then proceeded to analyze several of the objectives 
in detail and to suggest means for their achievement. 

THE NEED FOR A DEPENDABLE EVALUATION SYSTEM 

These sessions clearly revealed the community's desire for more informa- 
%on^bout their schools. All agreed that desegregation, altered social attitudes 
and shrinking financial resources would inevitably continue to prodUG^changes . 
If these changes vyere indeed ineljjctable, the community wanted them based upon 
clear, current and thorough information. ^ 

The district made great changes based in part on community input received 

from the Redesign conferences. Including implementation of a district^wide 

reading program (MARK) which utilized reading specialists for training teachers, 

provided a full time language arts teacher for each school, developed self-help 

reading laboratories, established an Educational Learning Center to provide 

intensive reading instruction for children with educational disabilities, Educa- 

tional Support teams, working in the schools and at the Center, and Block and ' . 

4 

Mini grants for schools ar)d individual teachers. 
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But after having accomplished wnese dramatic changes^ straitened re- 
sources and inadequate evaluation instruments and procedures,, made the district 
unable \o measure adequately the effect of the new Redesign program or to identify 
its weaknesses, strengths, or components needing ^jurther research and develop- 
ment.' 

One year after Redesign the district once again faced,^ budget deficit. 
Once again they sought from the community, through a series of hearings, sug- 
gestions about what kind of economies and program reforms the public would 
support. Once again the repeated advice of residents, teachers and parents 
called for the superintendent to make the necessary decisions, but to make them 
not based on political appeal, or professional intuition but rather based upon 
hard objective data. It became evident again that the district needed an evalua- 
tion-information system that would provide the information needed by the district 
administration to make crucial .decisions affecting particular educational programs 
staffing pattern's and levels of pupil achievement. 

The participants in this Maxi practicuin, five of whom hold administrative 
positions in the City School District of New Rochelle, settled upon this need to 
develop a system which would provide information necessary to decision-making 
as a project worthwhile in its benefit both to the district and to themselves. 
They sought the approval of the district's superintendent and, securing it, pro- 
ceeded to v/ork in developing the desired system. ^ 

Tho first step was to find out as much about the district and the origins 
of its present difficulties as possible. Much of the foregoing in this introduction 
is the product of that effort in historical research. Next came an analysis of 
current attitudes and perceptions. ' , . 



In April 1372 New Rochelle participated in a district-wide needs assess-- 
ment in conjunctior' with Battell^i Institution of C.oiumbus, Ohio^, The Needs 

0 

Assessment Survey identified areas of concern as perceived by different seg- 

ments of the QomiTiunity. In developing^the Needs Listing the Battelle staff 

defined as a "need" any item 'vVith index of 1,0 or higher. The index is the 

difference between the mean scores under (a) "desired" and (b) "actual" for 

each question. Any item with an index of 1,5 or higher they considered a 
6 

"critical need." i 

In the Battelle Survey students seemed most concerned about the difficulty 
In expressing their ideas to school administrators and the Board. They were 
concerned too with the need for certain course offerings at all three Ip^veJs and 
with the degree to which staff members help each student become an effective 
human being. They expressed the desire to have a stronger voice in the formula-- 
tion of school rules and regulations and more freedom in the various student 
media. T|iey also saw a need for student self-evaluation and a method to do so. 

Teachers were most ,concerned about the area of evaluation both as a 
means for the improvement of the educational program and as it might lead to 
recognition of and reward for superior performance. Another area of teacher con- 
cern was the need for programs that would prevent learning disabilities from 
developing and that would deal with them at the point of initial diagnosis.. 
Teachers saw the need for effective planning in their courses and in the district's 
total prdnram. They spoke in favor of training programs in educatfpnal planning 
for board members and in the development of a worthwhile orientation program for 
nev/ teachers* They also sought participation in the development of the school 
budget . 
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The community and parents also expressed concern in the area of evalua- 
tlon as it regarded board and administrator accountability. They were concerned 
with improving auxiliary services, sought improvement in pupil guidance, and 
in extra-curricular actl'/ities , Community members sought greater individualiza- 
tion of instruction and more effective communication about board and schoci 
activities. 

Administrators spoke out clearly in favor of effective in-service training 
programs foi administrators (and for-the encouragement and resources to partici- 
pate). They saw need for a comprehensive and in-depth orientation program for 
board members. They indicated a need for systematic (cost/effective) evaluation 
of the educational program, use of results to improve the educational program, 
and use of follow-up information obtained from graduates. 

In suir.mary the Battelle findings confirmed repeated earlier indications:- 
there is high degree of concern and agreement among all groups for educational 

planning and evaluation. 

Redesign, budget hearings, the Battelle program - each time the public 
or the staff had an opportunity to express its perception of the district's needs - 
the same need was named: the need for a more complete system of collecting 
" relevant information and of presenting it in such a way that it would be useful 
m the making of decisions. And that is whajl' we, the Maxi participantsj set out 
to provide: a comprehensive system for collecting, storing and displaying 
pertinent dcita, a system that would providfe the means to make the myriad deci- 
sions required of a school system in the -urrent era, a total edu::ationAl evalua - 
tiCM system. 
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The components for such a comprehensive evaluation system, we specu- 
lated, might include: 

1. ACHIEVEMENT MONITORING DATA: Information to teachers on 
' student achievement of specified learning objectives; 

2. NEEDS ASSESSMENT DATA: Information from teachers, parents and 
students, concerning the needs for achieving priority goals; 

3. TEACHER EVALUATION DATA: Administrator and students' subjective 
perceptions of teacher effectiveness; 

4. ADMINISTRATOR EVALUATION DATA: Teachers' and students' sub- 
jective perceptions of administratQ|| effectiveness; 

5. ATTITUDE MONITORING DATA: Information on teacher, parent, 
student, and relevant ^ther attitudes toward school and the school 
environment; 

6* GOAL SETTING DATA: Information to administrators from parents, 
teachers, and students, concerning what the priority goals for the 
school district should be; 

7. PROGRAM MONITORING DATA: Information relating to the cgst, 
conduct and effectiveness* of programs, individually and in related 
combinations; 

8, PROGRAM PLANNING DATA: Projections of the social, demographic, 
economic, and educational forces opera tiri^ in the district. 

The participants in this practicum are convinced that the need for a 

\ 

comprehensive system of evaluation in the City School District of New Rochelle 
can convincingly be demonstrated, Givon the proximity of the subject district to 
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New York City, given its size, racial and socio-economic composition, its 
financial status, the fluid, emergent nature of its programs, the (Changing ex- 
pectations of its public and the incredibly rapid technological and philosophical 
revolutions in the field of education, this district (and, we would posit, most 
others as well) needs a system^for collecting, for storing and for retrieving use- 
ful information about itself so that intelligent, informed decisions can be made 
relating to retaining, modifying or terminating components of the district's total 
operation. 

/ 
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CHAPTER II 

PROCEDURES AND PRACTICUM PARTICIPANTS' TASKS. 

! 

The order in which this Practicum was carried out was as follows: " 

Step I , ^ J 

\ 

A statement of need was developed by Mr. Olcott. This consisted of an 
analysis of the City School District's characteristics, its needs in terms of data 
for decision-making, and the purposes to be served by th'e installation of a 
district-wide evaluation model. \ 

^ - \ 

Step n. 

Each practicum participant fully investigated and prepared a description 
of an available evaluation model. 

a. Mr. Igidori and Mr. Zucker dealt with Stufflebeam's CIPP ModeU 

b. Mr. Po22i dealt with the Stake Model. 

c. Mrs, Gray dealt with the Provus Evaluation Model. ^ 

d. Mrs* Gess dealt with the Scriven Evaluation Model. 

e. Mr. Olcott and Mr. Samuels explored lesser-known evaluation model 
mentioned in the literature including those of Hatnmond, Tyler, Metfessel- 
Michaels and others, \ 

Step III, 

The practicum participants then developed the pro and con aspects of the 
model they investigated. They recommpnded those aspects of their model that 
were applicable and feasible in the subject district. 
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\ All practlcum participants , by using the results of their model analyses 
aid materials from other districts, conducted Joint lirain- storming and think- 
tank sessions. Through these sessions the practicum participants developed an, 
evaluation model suited to the subject school situation. 

Step V. ' ' 

Each practicuip participant developed training materials which related to 

'the model that he studied but which would have application in the n^w model. 

Step VI . • • • . 

The practicum participants , as a group, developed.a training program 

for various levels of school personnel within the district. 



- . _ • Step VII. - - ; ' 

These training programs were conducted for school building staff, school 

administrative staff, district administrative staff and special services staff 
according to the following schedule: 

a. Mr. Zucker conducted in-service training for all teaching staff 
within his elementary school building in the City School District of Mount Vernon, 

b. Mrs. Gess conducted in-service training for all teaching staff within 

her elementary building in Spring Valley. 

c. Mrs. Gray conducted training scssicns for department chairmen in 
her Junior high school in the City School District of New Rochelle. 

d. Mr. Pozzi conducted training sessions for elementary principals in 
the City School District of Now Rochelle. 

er|c ^^0 
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e. Mr. Samuels conducted training sessions for secondai7 principals 

and Mr. Olcott for district-widfe administrators includii^^ the superintendent and , 

his cabinet. " . • 

f. Mr. Isidori conducted training .assions for special service personnel. 

•» . "* 

Step VIII. 

Practlcum participants, concurrently with the training sessions incorpo- 
rating suggestions made by the trainees themselves concerning their emerging 
new roles, developed new role descriptions and functions for present staff. 

Step IX . ' . 

Practicum participants provided additional training sessions for key 

personnel, in their new -evaluation roles; 
SteeJC. 

Practicum participants developed guide books which contained procedures 
and\forms necessary to^onduct an evaluation of any type or level by.teachers, 
administration or community. 

Step XI. 

Items VII - X above constituted, in fact, the first steps in the direction 
of the implementation of an evaluation model on a district- wide basis. 

Step XII. 

Evaluation of a building program was implemented in an elementary school 
in Mount Vernon and Spring Valley using the newly developed system. 



Step XIII. 

Evaluation of a distrlct-wlde program operating In New Rochelle, using 
the newly developed system. 

ROLES OF PRACTICUM PARTICIPANTS 

Seymour Samuels 

I coordinated all aspects of the pract'icum. (50 hours) 
I investigated and prepared descriptions of less well known evaluation 
models. This investigation was done from the literature, visitations and con- 
sultants. (48 hours) . 

miter describing these evaluation models, I prepared pro and con aspects 

of the models. (32 hours) 

I made recommendations as to which aspects of the investigated models 
would be feasible and applicable to the subject school situation. 

I coordinated the practicum participants in brain-storming and think-tank 
sessions in the creation of a new evaluation model. (30 hours) 

I developed training materials that related th the models I investigated 
and which seemed to have application in the new model. (32 hours) 

I coordinated the practicum participants, in working sessions, in the 
creation of a training program for various levels of school personnel,, within the 

district. (16 hours) 

I conducted training sessions for secondary school principals in the City 

School District of New Rochelle. (30 hours) 
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I coordinated the practicum participants in the development of new role 
descriptions and operations for present staff and personnel. (16 hours) 

I coordinated the practicum pfirticipants in providing additional training 
sessions for key personnel in their new evaluation roles. (8 hours) 

I'coordinated the practicum participants in the development of evaluation 

guidebooks. (48 hours) 

I coordinated the initiation of an evalue^tion for a district-wide program , 
in conjunction with Pozzi, Olcott, Gray, and Isidori. (32 hours) 

I coordinated the efforts of Mount Vernon and Ramapo elementary schools 
irtth Mr. Zucker and Mrs. Gess. (10 hours) 

I coordinated evaluation of the practicum. (20 hours) 

I 

I 

i 

Richard Olrott 

. --I developed-the. Needs section of the practicum. (48 hours) 

I investigated and preparecl descriptions of less well known evaluation 
models. This investigation was done from the literature, visitations and con- 

sultants. (48 hours) 

After describing these evaluation models, I prepared pro and con aspect; 

of the models. (32 hours) 

I made recommendations as to which aspects of the investigated models 
would be feasible and applicable to the subject school situation. 

I participated in brain-storming and think-tank sessions in the creation 
of a new evaluation model. (32 hours) 

I developed training materials that related to the models I explored and 
which seemed to have application to the new model. (32 hours) 
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I participated with other practicum participants, in working sessions, .. 
in order to create a training program for various levels of school personnel 

within the district. (16 hours) 

I conducted training sessions for the superintendent and his cabinet,, 
in the use of the newly developed evaluation system . (30 hours) 

I participated w^th the other practicum participants in the development 
of new role descriptions and operations for present staff and personnel. ,My 
major emphasis was on evaluation roles for central office staff. (16 hours) 

I assisted the other practicum participants in additional training sessions 
for key personnel in tl^eir new evaluation roles. (8 hours) 

I assisted the other practicum participants in the development of evalua- 
tion guide books. My major emphasis was guide books for central administra- 
tion. (48 hours) 

In conjunction with Samuels, Pozzi, Gray and Isidori, I implemented the 
evaluation of a district-wide program. ('32 hours) 



Diane Gess 

I fully investigated and prepared a description of the Scriven evaluation 
model. This investigation was done from the literature, visitations and con- 
sultants. (48 hours) 

After describing the Scriven evaluation model in detail, I prepared pro and 
con aspects of the model.' I made recommendations as to which asp?^cts of the 
Scriven model would be feasible and applicable to the' subject school situation. 
(32 hours) 
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I Joined with the other practlcum participants in brain-storming and think- ^ 
tank sessions in the creation ofa new evaluation modeh I represented the Scriven 
model as well as an elementary principal in the development of this new model. 
(32 hours) 

I developed training m^ials that related to the Scriven evaluation model 
and which seemed to have application in the new model. (32 hours) 

I participated with the other practicum participants, in working sessions, 
m order to create a training program for various levels of school personnel within 

the district. (16 hours) 

I conducted training sessions in evaluation for the staff of my elementary 

school in Spring Valley. (30 hOCTs) 

I worked with the other practicum participants in the development of new 
role descriptions and operations for present staff and personnel. My major 
emphasis was on the evaluation role of teachers. (16 hours) 

I assisted the other practicum participants in additional training ses^sions 
for key personnel in their new evaluation roles. (8 hours) 

I assisted the other practicum participants in the development of evalua- 
tion guide books. My major emphasis was guide books for elementary teachers. 
(48 hours) 

' I implemented the evaluation of a program in my school building. (32 hours) 
Jerrold Zuckcr ^ 

I fully investigated and prepared a description of the CIPP evaluation 
model. This investigation was done from the literature, visitations and con- 
sultants. (4 8 hours) - 
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After describing the CIPP evaluation model in detail, I prepared pro and 
con aspects of the model. I made recommendations as to which aspects of the 
CIPP model would be feasible and applicable to the suWect school situation. 
(32 hours) 

I worked with the other practicum participants in brain-storming and 
think-tank sessions in the creation of a new evaluation model. I represented 
the CIPP model as well as the elementary principal in the development of thi^ 
new model. '(32 hours) * .| 

I developed training materials that related to the CIPP evaluation model 
and which seemed to have application in. the new model. (32 hours) 

I participated with the other practicum participants, ^n working sessions, 
in order to create a training program for various levels of school personnel, 
within the district.^ (16 hours) 

I conducted training sessions in evaluation for the staff of my elementary 
school in the City School District of Mount Vernon. (30 hours) 

I participated with the other practicum participants in the development 
of new role descrij)tions and operations for present staff and personnel. My 
major emphasis was on evaluation roles for teachers. (16 hours) 

■ I a'ssisted the other practicum participants in additional training sessions 
for key personnel in their new evaluation roles . (8 hours) 

I assisted the other practicum participants in the development pf evalua- 
, • •• 

tion guide books. My major emphasis was guide books for elementary teachers. 

c 

(48 hours) 

' I implemented the evaluation of a program in my school building. (32 hours) 
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* ' I perceived myself as the implementator. It was my responsibility to 
\ • review and implement in my school building the evaluation procedures and 

techniques developed by the group. The feedback of the procldures and tech-' 
niques was used to increase the feasibility and practicality of the evaluation 
model bfeing designed. ij 




Ruth Gray . % 

I fully investigated and prepared a description of the Provus evaluation 
' model. This investigation vl^as done from the literature, visitations and con- 
sultants. (48 hours) 

After dejSgribihg the Provu^evaluation model in detail, I prepared pro 
and' con aspects of the model. I'made recommendations as to which aspects of 
the Provus model would be feasible and applicable to the subject school situa- 

tion. (32 hours) * 

I participated with the other practicum participants in bfain-stonning and 
think-tank sessions in the creation of a neW evaluation model. I represented 
the Provus mdO^eLas well as department chairmen in the development of this new 

model. (32 hours) . ' , .. ^ 'i 

I developed training materials that related- to the Provus evaluation model 
and which seemed to have application in the new model. (32 hours) 

I participated with the other practicum participants, in working sessions, 

in order to create a training program for variouil, levels of school personnel within 

\ 

the district. (16 hours) 

Tconducted training sessions in evaluation for department chairmen In 

the City School District of New Rochelle. (30 hours) 

\ 

a . 
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i participated with the other practicum participants in the development 
of new tole descriptions and operations for present staff and cersonnel. My 
major emphasis was on evaluation roles for department chairmen. (16 hours) 

I assisted the other -practicum participants in additional training sessions 
for key personnel in their new evaluation roles. (8 hours) ^ 

I assisted the other practicum participants in the development of evalua- 
tion guide books. My major emphasis guide books for department chairmen. 

(48 hours) ' *• / 

In conjunction with Samuels, Olcott, Pozzi and Isidori, I implemented/ 

the evaluation of a district- wide' program. (32 hours) ^ 

I maintained a record of all our planning and working sessions (this in- 
cluded recording problems encountered; modifications made). Provided a basis 
for control by continuous monitoring (were we doing what we said we were going 
to do) and g basis for change by identifying our unmet needs and unused oppor- 
tunities; and stored Information for further use. This record served as the rough 
draft of this report for it chronicled bur practicum activities in detail. (50 hours) 

Facilities available to me f"-"" individual work included tape and tape 
recorder, some secretarial time, clerical materials, portable video-pak (for use 
during in-service training sessions), space in my office for storage. For group 
work, I had available transparency making devices, video tape, and hardware 
for in-service training. 
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Joseph Isidori ~~ 

I fully investigated and prepared a description of the CIPP evaluation 
model. This investigation was done from the literature, visitations and con- 
sultants. (48 hours) 

After describing the CIPP evaluation model in detail, I prepared pro and 
con aspects of the model. I made recommendations as to which aspects of .the 
CIPP model would be feasible and applicable to the subject school situation. 
(32 hours) 

I participated with the other practicum participants in brain-storming and 
think-tank sessions in the creation of a new evaluation model. I represented 
the CIPP model as well as the elementary principal in the development of this 

new model. (32 hours) 

I developed training materials that related to the CIPP evaluation model 
and which seemed to have application in the new model. (32 hL^r^ 

^ I participated with the other practicum participants, in working sessions. 

In order to create a training program for various levels of school personnel within 

the district. (16 hours) 

I conducted training sessions in evaluation for special service personnel 
in the City School District of New Rochelle.' (30 hours) 

I joined with the other practicum participants in the development of new 
role descriptions and operations foe present staff and personnel. My major 
emphasis was on evaluation roles for special service personnel. (16 hours) 

I assisted the other practicum participants in additional training sessions 
for key personnel in their new evaluation roles . (8 hours) 

ER?C :'9 
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I assisted the other practlcum participants in the development of evalua- 
tion guide books. My major emphasis was guide books for special service 

personnel. (48 hours) 

in conjunction with Samuels, Olcott, Gray and Pdzzl, I Implemented the 

evaluation of a district-wide program. (32 hours)' 



John Pozzl . .. / 



I fully investlg^ed and prepared a description of the Stake evaluation . 
model. This investigation was done from the literature, visitations and con- 
sultants. (48 hours) 

After Vie scribing the Stake evaluation model in detail, I prepared pro and 
con aspects of the model. I made recommendations as to which aspects of the 
Stake model would be feasible and applicable to the subject school situation. 

(32 hours) i 

I participated with the other practicum participants in brain-storming 
and Lhink-tank sessions In the creation of a new evaluation model. I represented 
.the Stake model as well as the elementary principal in the development of this 

new model. (32 hours) 

I developed training materials that related to the Stake evaluation model 
and which seemed to have application In the new model. (32 hours) 

I worked with the other practlcum participants. In working sessions. In 
order to create a training program for various levels of school personnel within 

the district. (16 hours) 

. ■ I conducted training sessions In evaluation for 10 elementary school 
princip^s in the City School District of New Rochelle. (30 hours) 
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I participated with the other practicum participants in the development 
ot new role descriptions and operations for present staff and personnel. My 
major emphasis was on evaluation role of principals. (16 hours) 

I assisted the other practicum participants In additional training sessions 
for key personnel in their new evaluation roles. (8 hours) 

I assisted the other practicum. participants in the development of evalua- 
tion guide books. My major emphasis was guide books for elementary princi- 
pals. (48 hours) , , 

In conjunction with Samuels, Olcott, Gray and Isidori, I implefmented 
the evaluation of a district-wide program, {^2 hours) 
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CHAPTER III . ' ■ 



A REVIEW OF AVAILABLE MODELS 



VThen the participants agreed to develop, test and install an evaluation 
system for the City School District they made their initial work the simultaneous 
exploration of two questions. The first was "What kind oi system does the 
district, given its characteristics, require?" and the, second "What kinds of 
evaluation systems are available?" vThe aivswers to the first question have been . 
summarized above. The answers to ^je second, consisting of a review of avail- 
able models and an'assessment of their utiiil^ in the subject district, are to be 
founcj below. , , 

THE CIPP EVALUATION MODEL ' 

In late 1967 and early 1968 the decision was made by the Phi Delt^ " 
( 

Kappa's Research Advisory Committee and Board of Directors to explore the area 
of evaluation of educational programs. This was prompted by the Elementary and 
Secondary Education Act of' 1965 which promised to improve educational program- 
ming. However, it was apparent in 1966 and 1967 that people in the field of 
evaluation did not sufficiently understand the nature of program evaluation nor 
did they have the evaluation procedures and techniques necessary to measure 
the projected educational improvements. As a result the Phi Delta Kappa's 
Advisory Committee recommended^the establishment of a natibnal Study Committee 
on Evaluation which was assembled by Dr. Daniel L. Stufflebeam. The people 
who were on this committee came from the Evaluation Center at the Ohio State 



University, the-R. & D. Center on Evaluation at U.C.L.A. and EPIC, a Title III 
Center in Tucson, Arizona. The purpose of this committee was to come up 
with a description of the process of evaluation regarding the conceptual and 

4 

.methodological needs of the field. One of the major results of this committee 
was the development of the CIPP Evaluation Model.- 

The CIPP Evaluation Model was developed to provide timely information 
in a systematic way for decision making which is a jaroactive application of 
evaluation. The CIPP Model also serves the retroactive purpose of providing 
information for accountability. 

According to the CIPP Evaluation Model, evaluation is a systematic, 
on-going process. The process includes tliree ba'jic steps: the delineating of 
questions to be answered and information to be obtained, the obtaining of rele- 
vant information, and the providing of Information to decision makers for their 
use to improve ongoing programs. Finally, according to the CIPP theories, 
evaluation serves decision making. Four kinds of decisions are served by the 
CIPP Evaluation Model. Planning decisions determine objectives. Structuring 
decisions project procedural designs for achieving objectives. Decisions in exe- 
cuting chosen designs are Implementing decisions, and recycling decisions 
determine whether to continue, terminate or make changes within the project. 

The aforementioned decision types are served by four types of evaluation. 
Context evaluation provides information about needs, problems, and opportunities 
in order to identify objectives. Input evaluation provides information about the 
strengths and weaknesses of alternative strategies for achieving the objectives. 
Process evaluation provides information about the strengths and weaknesses of a 
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strategy during implementation so that either the strategy or its implementation 
might be strengthened. Product evaluation provides information for determining 
whether objectives are being achieved and whether the procedure employed to 
achieve them should be continued, modified or terminated. 

The CIPP Model answers four questions: What objectives should be 
accomplished? What procedures should be followed? Are the procedures work- 
ing properly? Are the objectives being achieved? 

In order to see the relative worth of this method of evaluation, especially 
for accountability purposes, it is necessary to describe the CIPP Evaluation 
Model in detail. 

Definition 

The definition of evaluation that most suitably conforms with the proce- 
dures' and methodology of the Context, Input, Process and Product Evaluation 
model is: 

Evaluation is the process of delineating, obtaining and providing 

8 

useful information for judging decision alternatives. 

The purpose of evaluation is to improve the decision making process by 
establishing an interface relationship between the evaluation types and the 
various levels of decision making. 

The process of delineating, obtaining and providing is intrinsic at each 
level of the CIPP Evaluation Model so that useful information can be set up in 
• an organized manner to establish weighted decision alternatives. 

There are terms in the definition of evaluation that have to be explained 
in order to fully comprehend the workings of the CIPP Evaluation Model. 
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■ Process is a continuous and cyclical activity which subsumes many 

modes and steps of operations. 

oDelineating involves the focusing of informational requirements to be 
served by evaluation through specifying, defining and explicating. 

Obtaining involves the collecting, organizing, analyzing and report- 
ing of information through such means as statistics and measurement.- 
' ^ Providing means setting up the data collected into systems or svilj- 

systems which would best serve the needs ^f the evaluator and the decision 
maker. 

Useful information requires predetermined relevance criteria estab- 
lished by both the evaluator and the decision^ maker. 

Information is both the descriptive and interpretive data collected 
about the entities (tangible or intangible) of a program. 

Judging is the assignment of weights in accordance with a specified 
value framework, criteria derived therefrom, and information which relates 

criteria to each entity being judged. * 

" Decision alternatives are a set of optional responses to a specified 

question. 



Decisions to be Served 

The CIPP Evaluation Model advocates the use of acquired data to service 
decisions. There are four basic types of decisions that can be served by evalua 
tion. These are planning decisions, structuring decisions, implementing deci- 
sions and recycling decisions. 



29*. , 

• Planning Decisions 

Planning decisions pertain to the major changes that are needed in 
a specific program. Planning decisions are necessary when a discrepancy is 
noted between what a program is intended to do and what Is actually happening . 

Planning decisions would be concerned with such questions as: 
Should program cbjectives be changed? What priorities should the program 
, serve?" What are the problems which are impeding the attainment of objectives? ,, 
What specific behaviors should the students exhibit after participation in the 

programs? ' 

In summary, planning decisions are mainly concef^ned with progranl 
objectives and the obstacles that are impeding the attainment of those objectives. 
Structuring Decisions 

Structuring decisions determine the means that will be utilized in 
^ a program to attain the objectives . Such variables as method, content, organiza- 
tion, personnel, schedules, facilities, budget curricula, human and material 
resources, and the level of student, staff, parent involvement are all examples 
of structuring decisions . 

Structuring decisions Include action to operatlonallze a program. 
For example, the allocation olf budget, the recruitment of st^ff and the orientation 
to the Intents of the activity, the needed materials and facilities, assignment of 
responsibilities, definition of roles, motivation for staff to prepare for new 
activity are all necessary means for structuring a program so that objectives can 
be met. 
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Implementing Decisions 

Implementing decisions are concerned with putting the resources 
to work. These decisions deal with the actual working thorough of the program 
(operational procedures). These decisions elicit such questions as: Shpuld 
the staff be retrained? Should new procedures be Instituted? Should additional 
resources be sought? Should responsibilities be reassigned to staff? Should 
the schedule be modified? Is additional inservice required? Are outside con- 
sultants necessary? 

Recycling Decisions 

Recycling decisions deal with the relationship between the quality 
of attainments and the objectives. Recycling decisions indicate whether a 
specific activity or program should continue, terminate, evolve or modify. These 
decisions involve choices of product control. These decisions are not neces- 
sarily made at the end of a cycle of a particular program. Recycling decisions 
are concerned with attainments at any point in a program. 

Questions that illustrate what is meant by recycling decisions are: 
Aie the students' needs being met ^rough program implementation? Is the project 
failing? Are we solving intended problems? Was the project worth the money? 
Was the progress significant enough to continue the project? Has the project , 
resulted in improved teacher competence? 

Evaluation Types 

Since evaluation and decision-making have a symbiotic relationship, there 
are four evaluation types that correspond to the aforementioned decision types. 
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Context Evaluation 

Context evaluation is the most basic type of evaluation. Context evalua- 
tion provides a rationale for the setting up of objectives. Context evaluation: 

1. Defines the relevant environment. 

2. Describes the actual and desired conditions. 

3. Identifies unmet needs and unused opportunities. 

4. Diagnoses the problems that prevent needs from being met. 

5. Describes the values. 

Th^ methodology of context evaluation consists of two modes: 

1. The contingency mode «;earches for opportunities and pressures out- 
side of the immediate system to promote- improvement within. . 

J- 

Examples of the contingency mode of methodology would involve: 
study visits to other systems; exploration of the research and development 
literature; outside consultants; brainstorming'^retreats; assessments of com- 
munity values, attitudes and priorities, etc. 

2. The congruency mode compares actual and intended system per- 
formance. This mode is Concerned -.vith a school system's statement of goals 
and policies and the state's laws and policies governing education. 

An essential characteristic of this, mode of methodology is that it estab- 
lishes baseline data about the performance of a program. Through 
ology discrepancy information can be reported to administrators so that unmet 
needs can be taken care of, i.e. standardized test performance, lack of educa- 
tional opportunities for children from low socio-economic backgrounds , high rate 
of failure, negative student attitudes toward a particular area in the curriculum, 
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evidence of narcotics, malnutrition, high dropout rate, lack of parent particlpa- 
tlon . 

School systems should maintain context evc^uatlon mechanisms that 
provide both congruence and contingency data* 

Input Evaluation 

The purpose of input evaluation Is to determine what resources are 
necessary to accomplish program objectives. This Involves the Investigation 
of strategies and designs needed to attain the Intended results. 

The end product of Input evaluation Is an analysis of one or more pro- 
cedural designs In terms of potential costs and benefits, 'input evaluation would 
encompass the following processes: 

1. Alternative designs are assessed concerning time, staffing, budge^^^ 

requirements . 

2. Potential procedtiral barriers are Investigated. 

3. Costjof resources and strategies are studied. 

4. The relevance of design to program objectives Is also looked at; 
Input evaluation also provides Information that will determine If It Is 

necessary to use outside help to assist in the attainment of program objectives. 
Input evaluation is essentially ad hoc and mlcroanalytlc In comparison 

with context evaluation which Is mainly systematic and macroanalytlc. 

The methodology of input evaluation varies with. the amount of change 

that is necessary. For Instance If the change required is large and there is little 

Information available, then the Input evaluation will be broader In scope. 
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Context evaluation determines the amount of change necessary and th^ 
amount of information available.' If the context evaluation reveals that there i5 
need Ivr broad changes- and the information" available to make.those changes is 
low, then the input evaluation will be quite extensive. 

Input evaluation is also operative v.'hsn small program changes are neces- 
sary. Process evaiuftion may require slight changes in the operation of the' 
program. Thts may involve the investigation of additional resources to make the 
''changes. Hence input evaluation would once again becpme active. . 

* ' ♦ ^ 

Obviously, input evaluation can be either simplejor complex depending. 
/ ■ ' ■ 

upon the change setting (homeostatic, incremental or nepmobilistic) . 

Process Ev»klation ^ S ^ ^ 

Onc6 objectives have been formiilated based upon the context evaluation 
and resources are established through input evaluation for program operation, 
process evaluation occurs at that point when the program begins. Process evalua- 
tion .provides periodic fp^dback to persons responsible for implementing plans 
and procedures. 

Three Main Objectives of Process Evaluation 

1. Detecl: or predict defects in -the program design during the implementa- 
tion stages. ^ * ' * ^ 

2. Provide Information for programmed decisions. 

3. Maintain a record of the procedure as It occurs. 

The first objective may include interpersonal relationships between the 
students and staff, communications, logistics, understanding of program resources 
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and designs by staff, adequacy of resources, physical facilities, staff and ' 
time schedules. 

The second objective* involves projectivity and servicing pre-programmed 
decisions to be made by project leaders during the implementation of a program. 
If the input evaluation has been done properly, the evaluator should have little 
difficulty ia delineatiij^g pre-programmed decisions. 

The third objective of process evaluation is to note the main character- 
istics pf the project design. This would include concepts to be taught and the 
amount of discussion to take place, and a description of what actually takes 
place.' This information is necessary because it will be useful at later stages 
of the project when there is a determination of why objectives were or were not ^ 
achieved / 

Four essential elements of process evaluation are the provision for a 
full time process evaluator, instruments for describing the process, reguiar 
meetings between the process evaluator and program or project personnel, and 
frequent updating of the process evaluation designs. This is especially true in 
incremental and neomobilistic settings. 

The process evaluator should meet periodically with the project decision 
makers, but at these meetings the operations manager should discuss the project 
with the other program personnel to identify concerns and issues on future pro- 
ject activities. While the process evaluator should remain silent and listen, he 
should also provide informatiorj, about past operations to help identify issues con- 
cerning future operations. After the decision personnehhave identified the issues 
and questionCjthe process evaluator should again provide whatever relevant 
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informatfon he has. During pmnass evaluatir»n feedback sessions, the decision 
makers should define the conditions of relevajice for the feedback of information, 
and the evaluator should provide information within that framework. The evaluator 
should also help program personnel identify the questions which need to be 
answered at the next ^edbacl^essions; he might do this by summarizing the 
data collection plan for the imirlfediately intervening period and asking the decision 
makers if it will provide th^m v/ithHSie information they will need. 

Process Evaluation Methodology 

The methodology for process evaluation consists of be informal and 
fomal procediires • interaction analysis, open-end and end-of-the-day reaction 
sheets, interviews, rating scales, diaries kept by project personnel, semantic 
differential instruments, records of staff 'meetings > and suggestion boxes. 

. Relationship of Process Evaluation to Other Types of Evaluation 

Process evaluation is a function of the extent to which context and input 
evaluation have been performed adequately • the more adequate the context and 
input evaluation, the more certain the program director can be of how well his 
design v^ll operate and the less critical Is the need for process evaluation. 

When the rationale for the given objectives and project design is vague 
the project is headed for trouble and maybe failure. Therefore, the feedback 
.Uirough prucess evaluation is absolutely essential. 
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Product Evaluation ' ~ • 

The fourth type of evaluation Is product evaluation. Its purpose Is to 
measure and interpret attainments not only at the end of a project cycle, but as 

i 

often as necessary during the project term. 

The methodology of product evaluation inc^Gdes devising operational 

definitions of objectives, measuring criteria associated with the objectives of 

■ihe activity TTJOmparing these measurements with predetermined absolute or rela- 
tive standards and making rational interpretati<>us of" the outpomes using the 
recorded context, input and Irocess information. , 

Both context and product evaluation assess the extent to which ends are 
being attained. Context evaluation does this systematically v/ith respect to a 
total system, and product evaluation does so with respect to change efforts 
within the system. Context evaluation provides the specifications in terms of 
which product evaluation is later carried ouc. 

Input and product evaluations are easily distinguishable, for input evalua- 
tion occurs prior to the operation'alization of a change project, and product 
evaluation occurs prior to the operationalization of a change project, and product^ 
evaluation occurs during and after the project. Whereas context evaluation de- 
termines the specifications for%roduct evaluation, input evaluation provides the 
specifications for process evaluation. A major step within the input evaluation 

' phase is assessing the appropriateness of alternative and product evaluation 
designs that could be implemented as part, of a designed procedure. 

Process evaluation makes it possible to determine if actual procedure is 
discrepant from design, and product evaluation assists in determining whether 
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objectives are being attained. Together, both kinds of information provide a 
stronger rationale tha^ either one alo. a to judge whether a procedure should be 
continued as is, modified, or completely recycled. 

Strengths 

It is^ the judgment of 'the participants that the CIPP Evaluation Model 
with modification can be installed in the subject school district. However, it 
is important to note the strengths and limitations of the CIPP Model. 

/ The decision making process is central to administrative procedures. In 
many cases^ there is no systematic attempt in schools to arrive at sound decisions 
Decision makers must know what alternatives are available and be capable of 
making sound judgments about the relativ,e merits of the alternatives. Evaluative 
information is necessary because under the best of circumstances, judgmental 
processes are subject to human bias, prejudice and vested interests. 

One of the major advantages of in|talling the CIPP Evaluation Model in 
a school district is the offsetting of capricious, biased, decision making. The, 
CIPP Evaluation Model if properly followed v.-jll improve the quality of- decisions 
and hence the quality of programs. The Model will serve the decision makdr by 
identifying alternatives and by making valid and reliable information pertaining 
to alternatives accessible. c 

The different levels of evaluation and the counterpart.decision types help 
improve the direction of the school district. 

The Model is purpose-oriented. This is sound practice in administrative 
procedure. Each evaluation type irfthe CIPP Model has strategies, objectives, 
methodology and a relationship to decision making. 



The CIPP Model provides ongoing evaluation (formative) while the pro- 
gram is operative. This li essential so that changes and adjustments can be 
made while a program is going on. 

The CIPP Evaluation Model contains all of the essential characteristics 
of a logical structure of an evaluation design. 

These are as follows:, ^ 

F ocusing the Evaluation 

1 . The major levels of decision making to be served are identified. 

2. The CIPP Model allows for the projection of decision to be served. 

3. The CIPP Model defines criteria for each decision situation by 
specifying variables for measurement and standards for usq^in the 
Judgment of alternatives . 

\. The CIPP Model defines the policies within which the eval^tion 

\ \ must operate. 

\ \ 

\^ 

Collection of Information 
The CIPP Model: 

1. Specifies the source of the information to be collected. 

2. Specifies the instruments and methods for collecting the needed 
information, 

3. Specifies the sampling procedure to be employed. . .v 

4. Specifies the conditions and schedule for information collection. 
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Organization of Information 

The CIPP' Model specifies a format for the information to be collected 
and a means for coding, organizing, storing and- retrieving information. 

An alysis of Information 

The CIPP Model specifies the analytical procedures. 

Reporting of Information 

The CIPP Evaluation Model defines the audiences for the evaluation 
reports and specifies means^ for providing information to the audiences. 

Administration of the Evaluation 
The CIPP Model: 

1. Summarizes the evaluation schedule. 

2. Defines staff and resource requirements and plans for meeting 
these requirements. 

3. Specifies the means for meeting policy requirements for conduct 
of the evaluation. 

4. Evaluates the potential of the evaluation design for providing in- 
formation which is valid / reliable, credible, timely and pervasive. 

5. Specifies and schedules means for periodic updating of the evalua- 
tion design. 

6. Provides a budget for the total evaluation program. 

Another strength of the CIPP Model is that it has built into it a method 
of accountability. 
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Accountability is established by the CIPP Evaluation Model through the 
following means: \ 

!• Recording objectives and bases for choice through context evaluation. 

2. Recording of chosen strategy and design and reasons for their choice 
through input evaluation. 

3. Recording of actual processes through process evaluatior • 

4. Recording of attainments and recycling decisLions through product 
evaluation. 

Limitations 

>The CIPP Evaluation has obvious limita^tions that must be considered 
prior to the Installation of such a process. 

The cost for conducting this type of evaluation in a school district would 
be quite high, particularly if new, rather than retrained existing staff, are hired 
to implement it. The initial outlay by school districts for such a service, de- 
pending upon the amount of effort, coul^S be very costly. 

With modifications to serve local objectives and with a reorganization 
of the subject district that would make possible the employment of existing staff 
to conduct the evaluation, the CIPP Model could be installed in the district. 

DISCREPANCY EVALUATION MODEL (DEM) 

The DEM Is based on the work of Malcom M. Provus, Director of the • 

Evaluation Research Center, University of Virginia. The design defines evalua- 

9 

tion as "the comparison of performance against a standard." The model builds 
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in at each stage performance information to compare with a prior established 
10 

standard. Discrepancy is ascertained by "Judging the adequacy of that 
11 

performance . " 

Provus states that "both administrators and researcher s/evalua tors must 

see evaluation as a continuous management process which serves program im- 

. 12 

provement as well as program assessment purposes." ^ 

The DEM relies heavily on a team approach which synergisticall^com- 
bines the functions of evaluation, program planning, and program operation to 
create one self-improving system supported by professional consultation (the 
program staff and manager) and technical assistance (evaluating staff and 
researchers) to address the multiple needs of new education programs. The 
model calls for continuous feedback mechanisms in meetings of the team. 
The major purpose of this team then is to: 

1. Insure that standards acceptable to the decision-maker have.been 
defined. 

/ 2. Compare the practices, strategies and methods in the program that 
^ are to be eva^u. i against those predetermined standards. 

The DEM defines evaluation as a system for comparing performance against 
standards. Evaluation is a construct for improving educational programs. It is 
used to improve programs, with teachers and staff playing a key role in its con- 
duct. 

Hence, evaluation is a decision-making process that generates standards, 
modifies behavior, and, in short solves problems that are subject to continuous 
redefinition. 
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The pvurpose of this model is to analyze the interacting parts of an edu- 
cational system's performance; to provide information on program effectiveness 
to administrator (hereafter called decision-maker) who must make the decisions 
to keep, modify, or discard programs in operation. An administrator who has 

used the DEM suggests that the system is effective because it answers six 
13 

general questions: 

1. Does this program fulfill an actual need? 

2. Is it feasible to implement as designed? 

3. Is its operation as efficient as possible? 

4. Does it achieve its objectives as stated? 

5. Is its cost in line with that of other similar programs? 

6. And in the event of negative replies to any of these questions, 
can reasons for failure be assigned? 

The key emphasis of this model is in identifying discrepancies between 
standards and perfom^ance using the team approach. The approach is a problem- 
solving one which consists of the following evaluation activities: 

1. Carefully defining goals or purposes; 

2. A basis for the selection of program; , , , 

3. A detailed description of program o' plan of operation; 

4. A determination that support requirements for the plan of operation 
actually exist (including adequately trained staff); 

5. A method of monitoring the installation and operation of the program; 

6. The establishment of specific criteria for determining if goals have 
been met; and finally, 
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7. A way of feeding back information about program performance to 
those responsible for its management so that corrective action may 
be te^en eith^to redefine the program or adjust performance. 
Highlighting this ei^hasis is the concept that "educational program will 
improve only if teachers, administrators, and students become involved in a 
comprehensive effort to review a^d improve their own work,"^^ 

Role of Evaluator in the DEM 

The DEM provides for a team api^roach* However it designates specific 
activities for the program staff and the evaluation staff of the following types: 



Evaluation Staff Activity 

Identify decision points in the entire 
evaluation process 

Establish and maintain an apparatus 
whereby staff may formulate standards 

Ensure the adequacy of standard^ 
through the application of explicit 
criteria 

Communicate statement of standards 
to staff 

Identify information needed, to 
compare performance with standards 



Design a method of obtaining pro- 
gram information 

Report standard vs, performance 
discrepancy 

Identify decision points in the 
problem-solving process 



Program Staff Activity 
Identify standards 



Find ways in which to reformulate 
standards, if necessary 

Find ways to resolve differences in 
standards used by program staff 



Identify information availablfe or attain 
able in order to compare performance 
with standards 

Provide information descriptive of 
program performance 

Choose between action alternatives in 
regard to discrepancy 

Identify kind of information needed to 
pinpoint cause of program performance 
deficiency 
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Locate information about cause of 
program performance deficiency 

Identify decision points in choosing Detail criteria used to identify cause 
criteria to be used for selecting of discrepancy 

••possible" and "best" corrective 

alternatives Identify available corrective alternatives 

Locate and synthesize information 
as requested 

Identify criteria underlying choice of 
"best" alternative 

^ Choose "best" alternative for corrective 

action 

Thus illustrated, by listing the functions side by side, the interrelation- 
ship and then the activity strengthen and reinforce decision-making at all 
points, ^ • * 

Though in many school districts the evaluation staff and the program 
staff are one and the same, the ideal of DEM is to have the hea<J of the evalua- 
tion team be independent of the program unit. He is viewed separately as a 
team member who aids program improvement and counsels administration. 

The DEM assumes that the origin of standards (objectives) is derived 

from experience, knowledge and value. Provus sees "value as the key factor. . • 

Value determines what portions of a vast potential knowledge will be used as a 

15 

standard and then confers authority upon that standard. " The evaluation team 

(evaluation sta^f and program staff) then agree on standards. Discrepancy 

\ 
\ 

' evaluation suggests that the values underlying the use of a standard be made 
explicit (obviously they vary from school community to school community) . Thus 
the standards for a, program take on a specific form: a delineation of resources; 
a statement of intended outcomes; and a description of how resources will be 
converted into outcomes. 
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At Its simplest level evaluation Is the. comparison of performance (P) 
against a standard (S) . 

Figure 1. 
Discrepancy 

Evaluation Is the Comparison of Standard and Performance 
Relationship to Declslon>Maklng: 

The evaluation staff in an interrelated manner (with program staff) collects 
Information essential to program improvement and notes discrepancies between 
performance and standards. Every question involves a criterion, new informa- 
tion, and a decision. Ongoing evaluation provides the new information. The 
effective evaluation plan guarantees that information provided to decision- 
makers is: 

!• related to their priority concents; 

2. presented at an appropriate time in their planning cycle; 

3. presented in a format con si stent, with their use of the information 
in decision-making^ • 1 

4. fitting to the decision-maker's frame of reference. 

The DEM gives the program director an opportunity to become aware of 
the new forces Influencing the operation of the program and provides him further 
with a legal charge for making decisions. As a result decision-makers are 
forced by discrepancy information to make a choice. They must decide either 
to bring performance up to standard or change the standard. They must either 
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exercise control over operation or change the plan of operatl^^n , Discrepancy 
evaluation serves two criticaLfunctions qf management: control and design 
as in Figure 2 be lev . • , 

Hgure 2. 



(Design) 



S T P 



(Control) 



Discrepancy = standard Compared to Performance 
STAGES OF DISCREPANCY EVALUATION 

■ t • 

The discrepancy model has five stages {see Figure 3 below). 

The purpose of Stage I is to develop the program design. When complete 
the program design specifies standards for the evaluation of input, p-ocess, and 
output in precise terms and provides the necessary measures or Instruments to 
assess, perform.ince relative to each standard. The program design is compared 
with a set of design criteria: -these include statements of ultimate program goaJs 
and of the action to be taken to achieve these goals. Other design criteria are 
internal consistency of the program design, compatibility of the program with 
other activities in the system, and comprehensiveness of the design. 

Once a design has been derived from program staff, activity moves 
toward making the Stage I comparison. The program design is assessed for 
comprehensiveness, internal consistency, and compatibility. In assessing the 
adequacy of program design, two basic questions relative to the criterion of 
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comprehensiveness are asked: \l) Is there specific and complete intormation 
for each element of the program desigh, and (2) Is the Information in usable 
form? In checking program compatibility, you want to know whether the program 
conflicts In any way with any other programs in the entire school system. 

Stage II compares the actual Inputs and processes of the program With 

' the program design. ' • 

Data ts collected by the evaluator on the Inputs to verify that selection 
criteria, staff requirements, and organizational support conditions are being 
met." Discrepancies will be reported to the program staff and may result in 
changes In the program or changes In the design. 

Data Is collected on each process specified in the program design to 
verify its presence In the program. At this stage, more concrete standards are 
developed for^rogram staff performance. 

The result of this stage will be one or more reports to the program staff 
comparing In detail each performance variable with its standard and reporting 
any discrepancies that are discovered. The basic purpose Is to verify that the 
program has been Implemented in accordance with the program d'=-slgn. _ 

Stage III evaluation provides program staff with an estimate of the effect 
of the process elements on the output elements as a function of time. For this, 
r'^''t>\ continuous measurements must be taken. In Stage III, the Initial effects of 

partial treatment are evaluated by analysis of interim data . The evaluation staff 
should collect data describing the extent to which buident behavior Is changing 
^ as predicted. As a consequence of this stage of evaluation, the program staff 
learns whether or not the Intermediate program objectives are being realized on 
target dates, and If not, why not. 
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V 

Stage IV calls for th^ types of designs we have long employed in educa- 
tional research. At this stage, many of the relationships' between treatment 
conditions and effects discovered in Stage III can be properly expressed as 
independent variables in the experimental design stage. 

The objective of Stage V is to determine the most effective allocation 
of resources. Cost-benefit analysis is the ultimate rational step in the process 
of program development and assessment put forth in the discrepancy model. 

Using the five stages 6f evaluation, those responsible for a program 
can make an early and reasonably accurate prediction of its success or failure. 
In Stage Iv if agreement about the nature and purpose of a program cannot be 
reached, it is unlikely that the program will be successful. At Stage II, if 
essential resources cannot be obtained, nor adequate substitutes found, it 
would be foolish to continue r At Stage -III, if critical processes; Such'^'S* " 
student-teacher interactions or stimulus-response connections, do not function 
as expected, they must be reviewed and modified until those transactions essen- 
tial to outcomes are achieved. It is not necessary for decision-makers to wait 
until Stage IV or V (product assessment or cost benefit analysis) ic coinuleied 
before judging a program. Inefficient programs that cannot develop from one 
Stage to another despite the commitment of increased resources, be these staff 
time and energy or real dollars, should very likely be terminated. 
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Stages 
•I 



II 



III 



IV 



Figure 3 . 
Stages of Discrepancy Evaluation 



Performance 

Program Design 
Input Dimension 
Process Dimension 
Output Dimension 

Program Operation 



Program Interim Products 



Program Terminal Products 



Program Cost 



Standard 



Design Criteria 



Program Design 
Input Dimension 
Process Dimension 

Program Design 
Process Dimension 
Output Dimension 

Program Design 
Output Dimension 

Cost of other programs j 
wttjh same product 



The DEM provides for team involvement so that the e valuator or evalua^ 
tors and the program administraiui are interdependent (teamed). The evaluative 
process shows through an analysis of performance, discrepancies at variance 
with not only the standard but also with the assumptions and values that give 
rise to the standard. Periodic feedback provides tools for improvement and 
assessment. 

The following analogy developed by Malcolm Provus shows the ^mportar.c 

of the inter-relatedness or the "teaming" of evaluation and program staff: 

A man (school administrator) decides to take a trip. He tirst chooses 
California as his destination (goal). Next he must decide which of a 
number of alternate means of transportation to choose (selection of a 
program to reach the goal). His decision will be based on a number of 
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considerations such as time, money, etc. He decides td go by car.' 
Before he begins driving he must plan the precise route th%t he will - 
follow (plan of operation), what roads he will travel, how far he 
expects to go each day, etc. He will also have to engage in a number 
of activities to prepare for the trip, such as making sure the car has 
gas and oil and that all the parts are in working order. He'll also 
- have to make sure that he has proper financial coverage (support func- 
tions) and if he is driving a standard shift car when he has always 
driven an automatic, he'ma^i-have to learn to drive the car (staff train- 
ing) . Once he has started the trip, he will want to know whether he 
is making satisfactory progress toward reaching California (monitoring). 
By reading the road signs he will know v\^etiier he is on the right road, 
and if he reaches his predetermined destination each day he will know 
that he ip on schedule (feedback) . If he sees a sign that tolls him he 
is on the wrong road, he can alter his direcUon and get back on course. 
If he does not reach his daily destination, he will know that he probably 
underestimated the time and will want to revise his estipiate. Unfore- 
seen events such as a car breakdown (akin to the thousand and one 
uncontrolled events that can happen in a school system) could put him 
behind schedule. Our driver will know whether or not he has reached 
California if after a period of time he sees predetermined cues such as 
signs, landmarks, etc. (success criteria). 

Consider nov/, what could happen if the man did not go through all 
th.? important steps. First, if he hadn't decided where to go (goal) , it 
is obvious that he never could have started. Again he could not have 
started without a precise map of the route (plan of operation) he would 
wander aimlessly. If he hadn't made adequate preparations, he would 
risk getting stranded on the way. Without his map, the road signs and 
his stopping point each day would provide meaningless information. 
And finally, without knowing What cues to look for he would never know 
whether he had reached California. Even if he reached California he 
could never adequately describe to anyone else how to get there in an 
efficient way. 

Suppose the evaluator is waiting in California for the man who never 
arrives. All the evaluator can do is note that the man has not arrived. 
He has no knowledge of whether the route the man took was one that 
didn't lead to California or something went wrong along the way. (It 
may be that it's just taking him a few more days due to car failure.) 
If the evaluator had gone on the entire trip he not only would know why 
the man hadn't gotten to California, but could have helped him correct 
any deficiencies alon^ the way and increased the probability that he 
would have reached his destination. 

Obviously, a method of evaluation- that "goes alongj' will be of more 
utility than one that simply looks to see if goals have been met. 



ERLC ^ 



51. 

"There is consicjerable evidence in both industry and education that only 

when the personnel responsible for conducting a program are Involved in its 

„17 

examination and revision will the program improve and endure. 

■ Thus the DEM developed by Provus insures coordination and cooperation 

between project and evaluation so that program staff and recipients of program 

services alike are knowledgeable about the purposes and procedures of the 

evaluations. They then may willingly contribute to program improvement through 

) 

evaluation. 

This feature is judged by the participants to be the most significant single 
part of the DEM and an indispensible element in any evaluation system developed. 
The ideal team membership for DEM would be structured as follows: 

1. Several non-directive evaluation specialists skilled in small group 
process work and ethnological techniques, each of whom has responsibility for 
project-evaluation management but all of whom rfiay team up to facilitate group 
work. 

2. One or more psychometrists familiar with a wide range of group 
cognitive and affective instruments and capable of rapidly designing ad hoc 
instruments . 

3. A research-design specialist capable of drawing carefully defined 
samples, designing experiments, and directing the statistical analysis of data. 

A. One or more technical writers familiar with educational "language" 
and evaluation concepts • 

5, A data-processing unit with the capacity for data storage, retrieval, 

and statistical analysis as directed. ^ ^ 

^8 
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6. Subject-specialist consultants. 

7. A status figure capable of communicating directly with the superin- 
tendent of schools and all program directors. 

ROBERT E. STAKE EVALUATION MODEL 

In an attempt to evaluate educational programs many evaluators have 
formulated models and frameworks within which to accomplish this task. 
Robert Stake has designed a model Inherent in which, is the describing and judg- , 
ing of a mass of data. Stake's model assigns the evaluator the task of showing 
the relationship of all data obtained to the improvement of the program under 
study. He further assigns the evaluator the task of judging whether the out- 
comes desired or expected have been achieved. Finally the model allows us to 
note the congruence between what was intended and what was observed. It is 
v^th the use of his data gathering plan then that Stake feels decision-making 

will be facilitated. 

Gathering data from several quite different sources in several quite dlf^ 
ferent ways is needed according to Stake to fulfill the need of describing or 
judging an educational program. Whichever the purpose, three bodies of informa- 
tion should be tapped. 

The evaluation report should help to distinguish between (1) antecedent, 
(2) transaction, and (3) outcome data. An antecedent is any condition existing 
prior to teaching and learning which may relate to outcomes. Transactions 
are the countless encounters of students with teacher, student with student, 
author with reader, parent with counselor, the succession of engragements which 



comprise the process of education. Outcomes as a body of information would 
include measurements of the impact of instruction on teachers, administrators, 
counselors and others. Outcomes are the consequences of educating immediate 
and long-range, cognitive and connative, personal and community-wide. 

The evaluator prepares a record of what educators intend, of what ob- 
servers perceive, of what patrons generally expect and of what judges value the 
immediate program to be. The record may treat antecedents, transactions and 
outcomes separately within the four classes identified as Intents, Observations. 
Standards, and Judgments. 

AccoFding to Stake, the eval-uator seems to be increasing his emphasis 
on a full description of any given educational program in addition to continuing 
what he considers to be the other basic act of evaluation - judgment. The judg- 
ments are based on a formal inquiry process including collecting the judgments 
. of others. 

Evaluators must seek out and record the opinions (judgments) of persons 
with special qualifications. The opinions though subjective can be gathered 
objectively, independent of the solicitor's opinions. Those groups whose 
opinions on education are important and relevant are: spokesmen for society 
at large, subject matter experts, teachers, parents, and the students them- 
selves. These are the judges to be heard says Stake. 

The evaluation of a school program, says Stake, should portray the 
merit and fault perceived by the well identified, involved and affected groups, 
systematically gathered and processed.- Therefore, judgment data and descrip- 
tion data are both essential to the evaluation of educational programs. To be 
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I 

fully understood, the educational program must be fully described and fully 
Judged . 

Rote of the Evaluator 

The evaluator is a specialist who concerns himself with collecting, 
processing and interpreting descriptive and judgmental data. In his observa- 
tions the evaluator observes in a direct and personal way and sometimes uses 
Instruments. He may give special attention to the measurement of student out- 
comes but he does not fail to observe the other outcomes, nor the antecedent 
conditions and the instructional transactions. When he selects the variable 
for evaluation he must make a subjective decision and he must obviously limit 
those to be studied. He should give primary attention to the variables specific- 
ally Indicated by the educator's objectives. He must designate additional 
variables and search for unwanted side effects and Incidental gains . 

The evaluator should, according to Stake, be able to compare the Intents 
and Observations and note the discrepancies and describe the amount of con- 
gruence. He should also be In search for the contingencies between the ante- 
cedents, transactions and outcomes which are the relationships that permit the 
improvement of education. To test the logic of an educational contingency the 
evaluator relies on previous experience and perhaps research experience with 
similar observables. 

Part of the role of the evaluator is to make known wh'ch standards are 
held by whom, in the evaluation. Standards vary from student to student, 
instructor to instructor, and from reference group to reference group. 

CI 
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Stake considers "goals." "objectives." and "Intents" to be synonymous. 
He feels that the Intents should Include the planned for environmental condl- 
Uons. the. planned for demonstrations, the planned for coverage of certain sub- 
ject matter as well as the planned for student behavior. To be included are 
effects which are desired, those which are hoped for, those which are anticipated 
and even those feared. This class of data includes goals and plans that others 
have, especially the students. Whatever the goals, we must when evaluating 
examine what teaching, as well as what learning, is intended. The responsibility 
for describing curricular objectives, according to Stake, is the evaluator's. He 
must continually ask the educator for statements of intent. Stake feels it is 
not wrong for an evaluator to teach a willing educator about behavioral objec- 
tives. Obtaining authentic statements of intent is a new challenge for the 

evaluator, says Stake. 

Stake provides in his plan reports resulting from descriptive and Judg- 
mental data (including recommendations) to various audiences. The Judgments 
are based on either absolute or relative standards. 

In Judging the characteristics of a program Stake says there are two 
bases: absolute standards reflected by personal judgments, and relative 
standards reflected by characteristics of alternate programs. Before making a 
judgment for reporting, the evaluator determines whether or not each standard 
has been met. To judge is to decide which set of standards to follow and to 
assign a weight or importance to each set. Rational judgment in evaluation is 
the decision as to how much attention is to be paid to each reference group 
(point of view) in deciding whether to take some action, says Stake. Relative 
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comparisons find standards taken from descriptions of other prograins. From 
relative and absolute Judgment an overall rating of merit can be obtained in 
making an educational decision and from this Judgment a recommendation can 
be developed. 

Educational evaluation has, according to Stake, its formal and informal 
sides. Informal evaluation is recognized by its dependence on casual observa- 
tion, implicit goals, intuitive norms and -subjective judgment. Careful study, 
says Stake, reveals informal evaluation of education to be of variable quality, 
sometimes penetrating and insightful, sometimes superficial and distorted. 
Formal evaluation as Stake sees It is recognized by its dependence on check- 
lists . structured visitation by peers, controlled comparisons and standardized 
testing of students. Some of these techniques have long' histories of successful 
use, he says. He feels there is a great potential contribution, to education of 
formal evaluation. He feels the potential could be realized in a number of ways 
Educators should, for example, implore measurement specialists to develop a ^ 
methodology that reflects the fullness, the complexity, and the importance of 
their programs. They should spell out more clearly antecedent conditions and 
classroom transactions and attempt to couple them with the various outcomes. 
There should be an effort made to measure the match between what an educator 
intends to do and what he does do. For the evaluation of curricula, attention to 
individual differences among students should give way to attention to the con- 
tingencies among background conditions, classroom activities, and scholastic 
outcomes. , * 
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Finally, the formal evaluation endorsed by Stake Includes the evaluation 
of Its materials. As stated before, the two basic acts of his formal evaluation 
are description and judgment. 

Constructs Proposed 

For any one educational program there are two principal ways of pro- 
cessing descriptive evaluation data according to Stake: finding the contingen- 
cies among (1) antecedents, (2) transactions, and (3) outcomes and finding the 
congruence between Intents and Observations. The relationship (contingency) 
among the (1) antecedents [conditions existing prior to teaching and learning], 
(2) transactions [countless encounters of student with teacher, student with 
student, parent with counselor, etc.] and (3) outcomes [abilities, achievements, 
attitudes and aspirations of students resulting from an educational experience] 
is sought by the evaluator that permits program Improveme^rt. In the search to 
see If what was Intended (antecedents, transactions, outcomes) actually happened 
and were realized we discover the amount or degree of congruence. The evaluator 
compares the Intents and Observations, mating the discrepancies and describing 
the amount of congruence. The contingencies and congruences are subject to 
judgment by experts and part;lclpants. The importance of non-congruence will 
vary with the different viewpoints . Perceptions of the importance of congruence 
and contingency must be attended to very carefully by the evaluator. 

Stake feels judgments must become an increasing part of the evaluation 
report, ^valuators must seek out and record the opinions of persons with special 
qualifications. The two bases of judging the characteristics of a program (1) with 
respect to absolute standards as reflected by personal judgments and (2) with 
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respect to relative stdnd^rds as reflected by characteristics of alternate pro- 
grams. Each set of absolute standards, if formalized would indicate acceptable 
and meritorious levels for antecedents, transactions and outcomes. Before 
making a judgment the evaluator determines whether or not each standard is met. 
The Judging act itself is deciding which set of standards to heed. More pre- 
cisely Stake says judging is assigning a weight, an importance to each set of 
standards. Rational judgment in educational evaluation is a decision as to how 
much to pay attention to the standards of each reference group in deciding 
whether or not to take some adm>inistrative action. 

Relative comparison is accomplished in a similar fashion except that the 
standards are taken from descriptions of other programs. The evaluator selects 
the characteristics to attend to and which reference programs to compare to. 
From relative judgment we ca-n obtain an overall or composite rating of merit 
(perhaps with certain qualifying statements) a rating to be used for an educa- 
tional decision. From this final act of Judgment a recommendation can be com- 
posed. 

Stake would have a panoramic view of what was being evaluated. His 
model allows us to gain from as many points of view as are significant the 
existing conditions, the countless encounters of all involved parties, and ±e 
outcomes or consequences. With his plan of contingency and congruence we 
<hever look at a segment in isolation; we always observe and consider relation- 
ships of all the parts. 

Stake has determined that full description and full judgment are intrinsic 
in all formal evaluation. Stake feels that if rational Judgments are to be «iade, 

! 
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and he emphasizes judgmental criteria as mandatory to making recommendations 
for decision-making, then Informal techniques must be abandoned. Also he notes 
that too often judgmental statements are made about educational programs with- 
out describing the procedures used in arriving at the judgments. 

To Insure the publlcness of evaluative statements, standards for making 
judgmental statements must be explicated. All evaluation studies are compara- 
tive in nature, in the sense that descriptive data are compared to either absolute 
or.relative standards. If these standards are not made public according to Stake 
the credibility of evaluative statements can be legitimately questioned. The 
studies must be objective, scientific and reliable so that they might clearly be 
explained to the various audiences who seek this assistance for decision-making. 
The providing of immediate relative answers to assist in this is what is really 
being sought and must be gained for effective and beneficial continuance of any 
evaluation study. 

Desig n 

The design for evaluation as proposed by Stake is very general in nature. 

Every educational program has a rationale though only implicit at times. It 

J 

represents the philosophic background and basic purposes of the program. The 
evaluator must constantly ask himself, says Stake, whether the plan (evaluation) 
^nstitutes a logical step in the implementation of the basic purposes. The 
statement of purpose may be difficult to obtain or the instructor may not be effec- 
tive at presenting it. _ 
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With the establishment then of a rationale clearly recognized by all, the 
implementation of the Description Matrix seeking the contingencies between the 
antecedents, transactions and outcomes and seeking further the congruencies 
between what was intended and vvhat was observed will occur. Secondly, the 
implementation of the Judgment Matrix will occur. Here the evaluator uses the 
established or agreed upon Standards as a basis for Judgments which Stake feels 
are intrinsic in the required formal model of evaluation. These judgments will 
be rel3tlve and absolute and an overall rating of merit can be obtained to be 
used in making an educational decision. 
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Figure 4, 
Description and Judgment Matrices 
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In the evaluation of an educational program we must establish asnearly^^ 

as possible what the rationale of the program is. Although implicit at times it 
does represent the basic purposes of any program. As the evaluator proceeds 
with the development of the Description Matrix and then to'the Judgment Matrix 
he will have to constantly remind himself of this rationale so that there is an 
implementation of the basic purposes. 

The Description Matrix includes the "Intended" and "Observed Antecedents," 
"Transactions" and "Outcomes." " Antecedents " are any pf the existing conditions 
prior to the teaching and learning which may result. The student's aptitude, 
previous experience, Interest and willingness are examples of the "Antecedents" 
which the evaluator will describe. " Transactions " Include the succession of 
engagements which occur between students and their peers, teachers and coun- 
selors. "Transactlons"are the dynamics Included in discussion, explanation, 
administration. " Outcomes " result from the educational experience and Include 
such things as students' abilities, achievements, attitudes and aspirations. 
"OutcoincG" include not. only those vvliich are evident but application, transfer 
and relearning effects. "Outcomes" are the conseqnaences of education - irrirne- 
diate and long range, cognitive and connative, personal and community- wide . 

"Antecedents," "Transactions" and "Outcomes," the elements of evalua- 
tion statements are shown in the matrices to have a place in both description and 
Judgment. To fill tf^se in, the evaluator will collect Judgments (community preju- 
dice, problem solving styles, teacher personality) as well as descriptions. It 
is also indicated that Judgmental statements are classified as general standards 
of quality or as Judgments specific to a given program. Descriptive data are 
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classified as intents and observations. The evaluator can organize his data- 
gathering to the ma(rlx^rmat. \ 

The evaluator can prepare a record of what educators intend, of what 
observers perceive, of what patrons generally expect and of what judges value 
the Immediate program to be. The record may treat antecedents, transactions, 
and outcomes separately within the four classes identified as "Intents," "Otr--- 
servations/' "Standards," and '^Judgments," 

The "Intents" include on the matrix planned for demonstrations, planned 
for coverage of certain subject matter, etc. , as well as planned for student 
behavior. The collection of "Intents" is a priority listing of all tfiat may happen. 

The "Observations" include a description of surroundings and events 
and subsequent consequences. These observations are done both in a direct 
c^nd personal way and with, the use of instruroents^ "pbservations" are always 
made with the statement of purpose (Rationale) in mind. 

The first two columns of the data matrix contain the descriptive data and 
the format for processing these data is represented there. The data for a curriculum 
are congruent if what was intended actually happened. To be fully congruent the 
intended antecedents, transactions, and outcomes would have to come to pass. 
The evaluator should be able to compare the cells containing "Intents" and 
"Observations" tomote the discrepancies and to describe the amount of congruence 
for that row. Congruence does not indicate that outcomes are reliable or valid 
but that what was intended did occur. The evaluator 's task is one of identifying 
outcomes that are contingent upon particular antecedent conditions and instruc- 
tional transactions. 



The contingencies and congruences identified by evaluators are subject 
to Judgment by experts and participants just as more unitary descriptive data are. 
The importance of non-congruence will vary with different viewpoints. Percep- 
tions of the importance of congruence and contingency deserve the evaluator's 
careful attention. 

Limitations and Contributions 

• The most unique aspect of the Stake Evaluation Model used for the col- 
lection of information is his Description and Judgment Matrices. These Matrices 
are used in describing and judging educational programs based on a formal in- 
quiry process. 

Although the Stake Model does present limitations such as: (1) the 
inadequate methodology for obtaining information, (2) cells of design matrix 
overlap, and the d'.stinctions are not clear, and (3) there is the possibility of 
internal strife as a result because value constructs within the program could 
vary so; it does offer strong contribu^ ^ns to the evaluation field. It does pro- 
vide a systematic method for arranging descriptive and judgmental data thus 
emphasizing the inter- ^nd intra-relationships between them. It also considers 
bc^th absolute and relative judgment. 

mickael scrivem's evaluation model 

Michael Scriven has outlined a method of evaluation v/hich is kno\vn in 
educaticral circles as the Scriven Model. 

This model essentially deals with two processes in the act of evaluating. 
It concerns itself with the beginning or foimative evaluation of a project which 
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^ Is referred to as the producer's portion of the process and summatlve evaluation 
which concerns itself with the consumer or product portion of the project. 

Scrlven has built upon the foundation laid by Cronbach to suggest dis- 
tinctions and procedures which are of great practical' v^lue for the evaluation 

specialist. He has elaborated on the functions of evaluation by noting that, 

r 

while evaluation can play many roles in education, the evaluation process has 
only one functional goal - that of determining the worth or merit of something. 
By making this distinction, Scriven has emphasized that no study of any program 
can be labeled as evaluation unless some judgment is made. 

Understanding the difference between the formative and summ^'ive roles 
will help the evaluator delineate those methods that may be appropriately used 
in any one evaluation study. 

\ It is worth nating that the methods required reliably to arrive at an over- 

'v 

all appraisal in the Scriven Model have by no means been fully specified. It 
is a loosely woven set of ideals that in practical application have not been fully 
realized at this point in the development of the evaluation process. Scriven 
relies on the use of experts in making judgments. 

Evaluation is described by Scriven at the methodological level as a 
gathering and combining of performance data with a weighted set of goal scales. 
At the sociological or pedagogical level he becomes concerned with possible 
roles of evaluation. 

Scriven held that evaluation should attempt to answer certain types of 
questions about certain entities. The entities arc the various educational "instr' 
ments, processes, personnel, procedures or programs. It is meant to establish 
and justify the merit and worth of the above. 
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The major activities of the evaluator consist in gathering and combining 
performance data with a weighted set of goal scales to yield either comparative 
or numerical ratings in the justification of (a) the data gathering Instruments > 
(b) the weightings and (c) the criteria selected. 

The role which evaluation plays varies. It may form part of a teacher- 
training activity/of the process of curriculum development, of a field experiment 
connected with the improvement of learning theory, of an investigation preliminary 
to a decision about purchase or rejection of materials; ;t may be a data gathering 
activity supporting a request for tax increases or jresearch support, or a pre- 
liminary to reward or punishtnierrrof people as in an executive training program, 
a person or a classroom. 

Role of the Evaluator 

The basic fact is tliat the evaluator, while a professional in his own 
field, is usually not a professional in the field relevant to the curricuiun being 
reformed or, if he is, he is not committed to theN^particuiar development being 
undertaken. The evaluator therefore must recognize as his responsibility the 
uncovering and formulating of a testable set of criteria for the subject Deing 
evaluated. Scriven holds the evaluator responsible for making these judoments 
through his role in looking at goals, judging their worth and determining their 
worth. Formative evaluators should be sharply distinguished from suminative 
evaluators with whom they may work in developing an acceptable sumrrative 
design. Scriven states that if such a distinction is rrade, it becomes pc ssible 
to retain the advantaqes of objective professional evaluation without disrupting 
team efforts. 
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This total examination by the evaluator of comparative merit is done 
through evaluation reports with judgments explicitly statpd for either producers 
or consumers. 

The goals set forth by the formative evaluation and the product being 
judged at the summative end must be, according to Scriven, judged on their 
merit and worth in relationship to the goals and objectives set forth by the 
system • ^ 

The evaluation must include, as an equal partner with the measuring of 
performance against intended goals, procedures for the evaluation of the goals 
themselves* 

Any curriculum project has some kind of general objectives at the very 
beginning. These are elaborated on and judged by the expert evaluator in rela- 
tionship to their worth dependent upon the needs of the situation. 

Relationship to Decision Making 

How the information produced by an evaluation study using any one 
approach is related to the decision-making-process is of key importance in the 
use of any evaluation process. In the Scriven model judgments are explicitly 
stated for both producers and consumers in evaluation reports and utilized in 
decision making. 

Types of Evaluation 

Scriven cites the following as types of evaluation: 

1. Formative - summative 

2. Comparative - non comparative 
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3. Intrinsic - payoff 

4. Mediated. 

Formative - to assist In developing curricula. 

Summatlve - to assess the merit of curricula once they have been 

developed and put on the market. 

V 

comparative - to look at judged merits of content matter and goals and^ 
evaluate them in relationship to tests that are based on comparative 
norms. 

Non-Comparative - to judge them on their own or absolute scale or 
scores. 

Comparative evaluations are often very much easier than non--compara- 
tlve evaluations because we can often use tests which yield differences instead 
of having to find an absolute scale and then eventually compare the absolute 
scores . 

Scrlven recommends an outside expert be responsible for choosing any 
Instrument. 

Intrinsic - this involves an appraisal of the instrument itself; in the 
analog this would involve the evaluation of the content, goals, grading proce- 
dures, teacher attitudes, etc. This is also called secondary evaluation. The 
criteria is usually not operationally formulated, and they refer directly to the 
Instrument itself but only indirectly to its educational effectiveness or results. 
Defenders of the method state this Is the only way values can be evaluated. 

Pay-off • this approach proceeds via an examination of the effects of the 
teaching instrument on the pupil, and these alone, and it more usually specifies 
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these rather operationally. It describes effects on teachers, parents, etc., 
and may also show relev§nt effects. It may involve an appraisal of the dif- 
ferences between pre and post tests, between experimental group tests and con- 
trol group tests . 

The appeal of pay-off evaluation to the defenders of it Is the correlation 
of the evaluation of goals and methods with the actual effects on students. 

Mediated - Scriven outlines the following guide for practical implementa- 
tion of mediated evaluation: 

A three track approach to realize the objectives of a curriculum project 
through use of a project team who are assigned the task of goal formulation. 

a) As the project develops, the goals formulated should be regularly 
re-examined and modified in the light of divergencies from them that have arisen 
during the developmental activities, where it is felt that these changes have led 
to more valuable goals. 

b) Work should begin on the construction of a test-question pool. 
Progress tests will begin and the items can be thrown into this pool. The con- 
struction of this pool is the constructioh of the operational version of the goals. 
It should therefore be scrutinized at the same time as re-examination of the 
more abstractedly formulated goals occur. 

c) At this stage work should begin in getting some external judgments 
as to the cohesiveness of the alleged goals, the actual goals and the test ques- 
tion pool. 

The Scriven Model involves dravving a distinction between goals (claims) 
and roles (functions) and involves several types of evaluation. 

ERLC ^ 
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In terms of goals, Scriven's evaluation attempts to answer certain types 
of questions about certain entities. The entities are the various educational 
Instruments (processes, personnel, procedures, programs, etc.). The types 
of questions include questions of the form: How well does the instrument per- 
form (with respect to such and such criteria)? Does it perform better than this 
other instrument? What merits, or drawbacks does this instrument have (i.e. , 
What variables from the group in which we are interested are significantly af- 
fected by its application) ? Is the use of this instrument worth what it's costing? 
These activities consist simply in the gathering and combining of performance 
data with a weighted set of criterial scales to yield eithier comparative or numeri- 
cal ratings and in the justification of (a) the data gathering instruments, (b) the 
weightings, and (c) the selection of criteria. 

The roles of evaluation according to Scriven, may be different according 
to the context of a particular' educational situation. It may form part of a teacher 
training activity, of the process of curriculum development, of a field experiment 
connected with the improvement of learning theory, of an investigation preliminary 
to a decision about purchase or reception of materials; it may be a data isi^thering 
activity, etc. 

Within the framework of the model are the underlying principals of forma- 
tive and summative evaluation; comparative and non-comparative evaluation; 
intrinsic and pay-off evaluation; and mediated evaluation. 
. ' Scriven has based his criteria for selecting and judging evaluation proce- 

dures on (1) the establishment of goals, (2) the determination of the worth of the 
evaluation, (3) the presence of those constructs which are valid for the evaluation 
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being done, and (4) the employment of both formative and summative evaluation 
as a wholistic approach to program evaluation. 

Implications for Design 

The theme which runs through Scriven's thesis is the concern initially 
for the evaluation of objectives as prerequisite for program evaluation. 

The process of looking at many factors and the valuing of or the making 
of value judgments on the merit of the objectives before a program can be 
Judged worthwhile is the basis for the scientific process employed to judge 
the mex^it and worth of the program. 

• The distinction made between the formative and summative evaluation 
gives a special implication for curriculum workers who must design a plan for/ 
evaluation. The taxonomy of the criteria for evaluation gives the evaluatojrii 
checklist with which to work. The following is a rather generalized skeletal 
framework of the Scriven taxonomy that could hold implication for design. 

1. Conceptual Description of Educational Objectives 

a . Knowledge of; 

b. Comprehension or understanding of; 

c. Motivation (attitude/values/affect); 

d. Non-mental abilities; 

e . Non-educational goals , 

2. Manifestation Dimensions of Critical Variables 

a . Knowledge 

b. Comprehension 



ERIC 



¥ 4 




:ttltude 



71. 



d. The non-mental abilities 

3. Follow up data on above, 

4. Secondary Effects 

a. Effects on teachers ' ^ 

b. Effects on teachers' colleagues 

c. Effects on other students . 



e. Effects on parents 

f • Effects on school or college 

g. Effects on the taxpayer . 

h. Sundry effects 
5, Values and Costs 

a. Range of utility 

b. Moral considerations 

c. Costs / 



6. Explanatory Evaluation (sometimes a' part of process research and 
should be secondary to previous kinds of evaluation,) 

Contributions 

The major contributions of the Scriven model are: 

1. The application of the discrimination between formative or on-going 
evaluation and summative or product evaluation, 

2. The focus on the direct assessment of the worth of the objectives as 
determining the merit of the evaluation. The concept of valuing as a prime con- 
sideration in the process of evaluating. 



Effects on administrators 
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3. The application of the many types of roles in diverse contexts in 
the process evaluating. 

4. The determination that although evaluation can play more roles, the 
prime purpose is to determine the worth or merit of something. 

5. The delineation pf the types of evaluative procedures and the purposes 
for each (summative-formative; comparative-non-comparative; intrinsic-payoff; 

and mediated). f 

6. The analysis of determining the credibility of the means and the 

justification of the' ends . ^ 
Li mitations 

1. The determination of the facets of the program, including all the 
variables for the student along with the costs and all other possibilities, in de- 
termining the worth of the program creates tremendous methodological problems. 

1. There has been developed no process for assessing the validity of 
judgments being made. 

3. There are several ambiguous overlapping concepts that have not been 
spelled out. 

4. The model relie? on the employment of outside experts rather than 
locdl staff already on hand. 

OTHER EVALUATIQN MODELS 
« 

In ac' ^on to the analyses made of the major evaluation models reported 
above, analyses were also made of other designs which appear less frequently 
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Tn\he literature. These include the work of^wton S. Metfessel and William B. 
Michael; Robert !. Hammond, and Ralp^t^ler. 

• NEWTON S. METFESSEL ANDlWaLIAM B. MICHAEL 
University of South¥rn California 

The thesis upon which Metfessel and Michael operate in their design |or 
evaluation is largely based on the Tyler model oi comparing measured performance 
with behavioral standards. Objectives are stated as operational de.finitions in- 
volving measurable and observable changes in behaviors that have been judged 
to be significant and relevant to the broad goals and the philosophy of the educa- 
tional institution under study. 

Definition 

Comparing measured performance with behavioral standards. Criterion 
measures are used in evaluation of the attainment ot objectives in school programs. 
Specific objectives both the cognitive and affective domains are stated. 

Purpose 

To formulate recommendations that furnish a basis for further implementa- 
tion, for modifications, and for revisions in broad goals and specific objectives. 
Metfessel and Michael feel that this may be of some help to teachers, administra- 
tors, counselors and consultants to public schools, and other profe^ional per- 
sonnel whose experience in evaluation may be limited. 

Key Emphasis 

Specification of objectives and using multiple criterion measures to 
assess outcomes. They key in on the involvement of many different audiences. 
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Their paper, "A Paradigm Involving Multiple Criterion Measures for the 

18 

Evaluation of the Effectiveness of School Programs/' lists eight major steps 
in the evaluation process. 



Types of Evaluation ^ ' 
In brief they afe capsuled as follows: 

1. jR^olvement of many people from all are^s of school and home com- 
munity. , ' 

2. Construction of a cohesive paradigm of broad goals and specific 
objectives. / 

3. Translation of specific/4)bjectives into communicable form. 

/ 

4. Development of neceySsary instrumentation for criterion measurement. 

5. Periodiq observations through instruments of behavioral measurement. 

6. Analysis of data. 

7. Interpretation of data in terms of judgmental standards and values. 

8. Formulation of recommendations. \ 

Role of the Evaluator 

The evaluator is a measurement specialist who Involves lay individuals, 
school people and students in developing a set of recommendations. 

The evaluator is familiar with both standardized and nonstandardized 
instruments and techniques to collect data on evaluation studies. 

Relationship to Decision-Making ^ 

Suggestions are made concerning revisions in objectives or program strate- 
gics. Feedback is given to all individuals involved in the school program. 



The specific behavioral ^jectives are translated into a form that is 
both communicable and applicable to facilitating learning. Recomm'^ndaticns 
can thus be made for furtherJjiiEl^ifientationT^ modification and for revisions 
In the broad goals and specific objectives ^that improvements can be realized. 

\ i 

Constructs Proposed ^ 

The system of multiple criterion measurement is set forth as follows: 

1. Measures which are indicators of status or change in Cognitive and 
affective behaviors of students in terms of standardized measures'and scales, 

2. Measures which are indicators of status or change in cognitive and 
affective behaviors of students by informal or semfformal teacher-madte instru- 
ments or devices. 

3. Measures which are indicators of status or change in student behavior 
other than those measured by tests, inventories and obsiervation scales in rela- 
tion to the task of evaluating objectives of school programs, 

4. Measures v/nich are indicators of status or ihange in cognitive and 
affective behaviors of teachers and other school personnel in relation to the 
evaluation of school programs . 

5. Measures which are indicators of community behaviors in relation to ' 
evaluation of school programs, 

implications for Design 

The eight stage evaluation process sets forth a model which, if imple- 
mented, would give a design involving a wide audience, and specification of 
objectives. The provision for periodic observations adds credibility to the 
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constant and consistent measurement using multiple criterioi). Provided also is 
the method for infer pre tacion and recommendation to the various audiences. 

Contributions 

The major contributions are judged to be in (a) the involvement of a large 
audience'; (b) the specification of objectives; (c) the continuous feedback and 
periodic observations; (d) tho laying out of the multiple criterion measurements 
for the user; (e) the inclusion of reporting to the various audiences as a vital 
part of the e\:ilual^on proqess; (f) using measurement and technology for objective 
data gathering; and (g) the ease within which the model may be applied. 

Limitations 

The following are limitations of the Metfessel, Michael Model: 

1. Too much focus on traditional, standardized instrumentation. 

2. Testing is excessive and thereby impractical. 

3. The focus is only on outcome. Little is done on process or ante- 
cedent evaluation. 

4. There is a lack of methodology for establishing standards, evaluating 
their relevance, or modifying them during the process. 

' ROBERT L. HAM M OND . 
University of Oregon 

The thesis developed by Robert Hammond''state3 that the success or 

failure of innovations *in rnodern ^)rograms of instructioh is determined by, the 

•A 

interaction of specific forces within the educational environment. He ci^es 
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these forces in terms of specific. dimensions and* variables which operate in a 

< 

three dimensional structure, the interaction of which produce combinations of 
effects which must be considered in tlie evaluation of a given program. The 
importance of any combination of these variables is determined by the nature of 
the program -selected for study. 

Definition 

I 

Hammond assesses the effJo iveness of current and innovative programs 
at the local level by comparing behavioral data with pre-set objectives. 

Once the forces effecting a given innovation have been identified and 
placed in a structure which permits an analysis of the interaction of these forces 
the next step is that of placing the structure in a working model for evaluation 
through carefully defined steps, 

Purpos^ ^ 

The purpose of this model. is ta find out whether innovation is effecti ve 
in achieving expressed objectives, 

Hammond states that once the objectives have been defined, it is up to 
the evaluation effort to measure the behavi^^de scribed in the objectives through 
analysis of collected d?.ta, | "\ 

• ' ( 

Key Erriphasls 
^ 

Hammond stresses the importance* of local program development in evalua 

tion. 
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He recommends that teachers and administrators attain skills necessary 
to evaluate Instructional programs • Once these skills have been developed, 
the school district should progress to the point that they can operate independently 
of any external evaluatlo*n support. 

Role of Evaluator 

The evaluator should be a consultant who provides expertise In data 
collection and trains local evaluators (program personnel^. 

The personnel might need to secure the help of measurement and analysis 
specialists but should also seek assistance from an evaluator v/ho would at the 
same time train them sufficiently to carry out most future steps in future evalua- 
tion studies unassisted. 

Relationship to £ bi ectives 

This evaluation focuses on the definition and measurement of behavioral 
objectives* 

Hammond places objectives stated in behavioral terms in the Behavioral 
DiTTiGnsion of his model. He recognizes the three variables of "Cognitive/* 
•'Affective/' aod "Psychomctor" behavi.^rs (perceptual variable is being studied). 
The study of a given factor is determined by time, availability' of tests and 
procedures, and the needs of a given school district. 

Relationsh ip to Decision Makin? 

Evaluation is the source on which to base decisions about instructional, 

• * 

institutional and behavioral dimensions. 
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Sound evaluation procedures require that the process bealn with a thorough 
examination of the current programs. Before attempts at innovation are made, 
adequate baseline data is required to make those decisions which determine 
the direction of the change process. 

Types of Evaluation ^ 

The Hammond model uses three dimensions for describing programs: 
(1) the Instructional Dimension; (2) the Behavioral* Dimension; and (3) the 
Institutional Dimension. 

Each diniension consists of a number of specific variables. 

The Instructional Dimension describes the innovation in terms of five 
such variables. The variables are: Organization, Content, Methodology, 
Facilities and Costs* 

Constructs Proposed 

This mode^ sets up the following constructs: 

1. The application of the kind of evaluation design to the existing 
program. 

2. Decisions about the adequacy of the current program in relationship 
to the objectives. 

3. Feedback from the decisions about the program adequacy leads to 
innovation. Is it necessary? 

4. Application oi the evaluation is studied in itG relationship to the 

evaluation.. \ 

5. Continuing feedback. 
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Implications for Design ^ 

The use of a multi-variate structure focusing on interaction ofdimensions. 
Both the variables and the dimensions are discussed within this report. This 
is shown as a systematic way to assess the effectiveness of both current and 
innovative programs. 

Contributions 

The outstanding contributions of the Hammond model are considered to be: 

1. The use of local personnel who can carry on the evaluation process 
once it has been initiated. 

2. The consideration of interaction of the dimensions and variables 
which constitute a total appraisal. ^ 

3. Emphasis on feedback in program development and revision. Self 
evaluation and rethinking of action is continuous, 

4. Specification of behavioral objectives. This, as Hammond states, 
represents one of the most crucial steps in the eva^^uation process. Properly 
stated objectives will: (a) specify the kind of behavior which will be accepted 
as evidence that the learner has achieved the objective; (b) state the conditions 
under v/hich the^ behavior will be expected to occur; and (c) specify the criteria 
of acceptable performance by describing hov/ well the learner must perforin. 

Limitation s 

There is difficulty quantifying data involving several of the dimensions 
ond variables. Additionally, the model, because of its intricacies, may be 
complex and time-consuming to set up. 
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The "cube" representing the dimensions and variables might become the 
source of evaluation rather than the objectives that are underlying. 
The model neglects the Importance of informed judgment. 
The motivation of local personnel is a problem unconsidered in Hammond's 

model. 

RALPH W. TYLER 

Tyler*s thesis states that the process of evaluation is essentially the 
process of determining to what extent the educational objectives are actually 
being realized by the program of curriculum and instruction. Since educational 
objectives are essentially changes in human beings or aimed at producing certain 
desirable changes in behavior patterns, evaluation is the process of determining 
the degree to which these changes in behavior are actually taking place. 

Definition 

Comparing student performance with behaviorally stated objectives. 

This is basically a congruence definition first proposed by Tyler as an outgrowth 
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of his work in the Eight-Year Study at Ohio State University. 
Purpose 

To determine the extent to which purposes of a learning activity are 
actually being realized by the program ot curriculum and instruction. 
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Key Emphasis 

There is a specifibation of objectives and a measuring of the learning 
outcomes of pupils. Since objectives are essentially changes in human behavior, 
the objectives aimed at are to produce certain desirable changes in the behavior 
patterns of students • 

Role of the Evaluator 

Tyler sees the evaluator as a curriculum specialist ',vho evaluates as 
part of curriculum development and assessment. 

r 

/ 

Relationship to Objectives 

The evaluation implies attainment of the behavioral objectives first 
identified at the beginning of the course or program. 

Relationship to Decision Making 

The gathering of actual pupil pe-^formance data will provide the informa- 
tion for the decision-maker to use to identify strengths and weaknesses of a 
course or curriculum. 

Types of Evalu ation 

Tyler uses the Pre- and Post-measurement method of evaluating the 
congruence of performance with objectives. 
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Constructs Proposed 

The following are those constructs proposed by Tyler in his evaluation 

model: 

1. Statements of objectives in behavioral terms. 

2. Teaching objectives are pupil-oriented. 

3. Objectives must consider pupils' entry behavior, analysis of our 
culture, school philosophy, learning theories, new developments in teaching, 
etc. 

Criteria for Judging Evaluation 

The criteria tor evaluation must rely on: 

1. Behavioral objectives that are clearly stated. 

2. Objectives that contain references not only to course content but 
also to mental processes applied. \ 



Implications for Design 

There must be interpretation and the use of assessment when designing 
program. The designs developed should assess student progress. 

Contributions 

The Tyler model contributions may be listed as follows: 

U It is ed-sy to. assess whether behavioral objectives are being achieved. 

2. It is easy for practitioners to design evaluative studies. 

3. It checks the degree of congruoncy between performance and objectives 
O and focuses on a clear definition of objectives. 

ERIC 00 



limitations 

There is a tendency to oversimplify program and focus on the terminal 
rather than on on-going and pre-program information. 

There is a tendency to focus directly and narrowly on objectives, with 
little attention to the worth of the objectives. 
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CHAPTER IV 
THE NEW ROCHELLE EVALUATION MODEL 

The evaluation model adopted by the practicum participants was 
developed from Stufflebeam's CIPP Model. In addition to the basic structure 
of the CIPP model, however, the New Roch^lle approach incorporated the con- 
cepts of a team and discrepancy evaluation of Provus, as well as the methods 
for the collection of descriptive and judgmental data emphasized by Stake, 
The model also contains elements derived from summative evaluation as 
described by Scriven and suggested the development of a new organization 
structure for the school district as well as new role descriptions for ^ taff , 
concepts implied in the writings cf Hammond and Tyler, 

The resulting eclectic menage is xhe framework which the participants 
believe will most effectively serve the subject district at the present time, 

THE MODEL 

The unifying theme for the New Rochelle Model is the following defini- 

19 

tion: 

Evaluating is the process of delineating, obtaining, and 
providing useful information for judging decision alternatives. 
This definition emphasizes that evaluation is a continuing process; that this 
process includes the three steps of delineating, obtaining, and providing infor- 
mation; and that this information should meet criteria of utility and should 
guide decision-making. 
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Given that evaluation supplies Information for decision-making/ the 
decisions to be served must be known. Decisions are divided Into four classes 
called planning, structuring, implementing, and recycling decisions. Planning 
decisions are choices of objectives. Structuring decisions are those in design- 
ing projects to achieve given objectives. Those required for operationalizing 
and executing a project design are implementing decisions. Recycling decisions 
refer to the judgment of and reaction to project results. 

« 

Figure 5. 
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A ; imework that Relates CIPP to the 
Formative-Summative Conception of Evaluation 



The CIPP aspect of the evaluation process includes the three main steps 
of delineating, obtaining, and providing. Delineacing the questions to be 
answered and providing obtained information to decision-makers are interface 
actlvitii's requiring collaboration between evaluator and decision-maker. The 
ol?taining of information is a technical activity involving measurement, data^ 



processing and statistics, and Is executed mainly by the evaluator. or in the 
case of the New Rocheile Model, by the evaluation team. The delineating, 
obtaining, and providing steps' provide the basis for the New Rochelle method- 

ology of .evaluation. 

Figure 6 is a framework for designing evaluation studies. It includes the 
dimensions of types of evaluation, use*? of evaluation, and the, steps in the 

evaluation process . " 

To use this framework, one should first determine what types of evaluation 
will be conducted and then for each selected type determine whether decision- <5 
making and/or accountability is/are. to be served. Then the delineating, obtain- 
ing and providing step.»-3houi'd be defined for each of the chosen columns in the 
matrix. The result of using this framework is a set of evaluation designs to be 
implemented. In general, such designs specify 'vhat questions will be addressed, 
how the needed information will be obtained, and how the information' will be 
reported to the designated audiences , 



Figure 6. 
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A Framework for Designing Evaluation Studies 



According to CIPP, evaluation designs and reports should be judged In 
terms of three standards. The first is technical adequacy and concerns validity, 
reliability, and objectivity. The second standard is utility and iovolvesj the,........* 

relevance, scope, timeliness, importance, pervasiveness, and credibility of 
the evaluation. The third standard is cost/ effectiveness > 

Evaluation has a b^asis in the need for information which requires the 
delineation of information ;needs, and has utility in making decisions which 
require providing information. Both delineating and providing useful information 
for judging decision alterna,tives require interaction with decision-makers and/or 
potential audiences. This interaction defines the interface of evaluators with 
other roles. Schematically, the interfaces appear as shown in Figure 7 below. 

Since there are four kinds of decisions, there are also four kinds of 
evaluation. C ontext evaluation serves planning decisions by identifying unmet 
needs, unused opportunities, and underlying problems. Input evaluation serves 
structuring decisions by projecting and analyzing alternative procedural designs. 
Process evaluation serves imols^menting decisions by monitoring project opera- 
tions. Product evaluation serves recycling decisions by Identifying and assess- 
ing project results . 

In addition to serving dec|sion-making, the Model provides ar ba^is for 
accountability. Conte^jt evaluation provides a record of objectives chosen, 
t^ho^ rejected, and the relation of chq^en and rejected objectives to infornfation 
about needs, opportunities, and problems. Input evaluation provides a record 
of the actual implementation process. Product evaluation rocords project attain- 
ments and decisions concerning the continuation, modification or termination of 
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the project. Overall, through recording information and the decisions influenced 
by the information, program managers can maintain a strong basis for account- 
ability. 

One way to summarize the CIPP aspects of the Model is the framework 
that relatei^ its four evaluation types to their two uses. Figure 5 shows that 
context, input, process, and product evaluation serve both decision-making and 
accountability. By equating ^^>hll^■atlort for decision-making to formative evalua- 
tion end evaluation for accountability to summative evaluation one can see in 
> 

this framework the relationship between the Scriven and CIPP conceptualizations 
of evaluation. The four kinds of evaluation are formative if they are conducted 
proactively to serve decision-making. They^re sumtn&tlve^if'th-eyaretsonducted 
retroactively to serve accountability. 

Evaluators are shown in interface with the many roles to be found operating 
in the educational community, ail members of which are potentially decision- 
makers. The process of delineating information needs requires the evaluator to 
work with decision-makers and members of the educational community. The 
evaluator will be establishing decision settings as well as the criterion variables 
that will be applied in the evaluation. 

At the opposite end of the evaluation process - that of providing informa- 
tton - are shoVn interfaces of the evaluator with the same multiple role types. 
The evaluator maySbe delineating information needs in relation to only one or two 
people, perhaps the superintendent and other administrators. He may then be 
reporting back to these same people - providing information. The evaluator will 
be determining, through interaction with many people, more broadly-based 
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information needs, and that his report will be made to^an audience much broader 
than the primary decision-maker. 
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Interfaces of ^valuators v/lth Decision-Makers and/or Audiences 
Figure 8 illustrates various methods which will be used by the evalua 



cor 



in delineating information needs, as -well as nun^.erous media or method-^ to be 
used in dissem^inating the infor-r.ation provided. 

Emphasis ir- pieced or. the delineation and providing of information because 
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• Interface Methodologies with Decision-Makers and/or Audiences 

The overall framework for'^he evaluation design Is based on the three 
major task areas of the definition, developed by Stufflebeam, - delineating, 
obtaining and providing. These three areas can'be broken down Into activities 
which will be related to the decision setting, the decision maker, and the type 
of evaluation required. These activities, within units, each have a series of 
tasks. Figure 9 depicts these units of activity and the tasks within. 
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The New Rochelle ©valuation model will involve the following steps - 
First, the evaluation team will delineate the activities that are to be evaluated, 
the decisions about those activities that are to be served, the information needed 
to service those decisions, and the policies that will govern obtaining and pro- 
viding the information. Second, they will obtain the needed information. Third, 
they will communicate this information to those who will make the decisions. 
ThYs process ensures that"a~ecisIohs are maae that can ferult in defensible goals 
and in activities which are efficient and effective in meeting these goals. 

Forming the basis tor the evaluation model proposed are the following 
eight premises: 

1. Since the purpose of evaluation is to provide information for dccision- 
maicing, it Is necessary to know the decisions to be served. 

09 
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1\ For evaluation to be relevant to decision-making, the evaluator must 
be oriented to the decisions to be served and function within this orientation. 

3. A valid evaluation model should be grounded in sound conceptualiza- 
• tions of the different change settings (homeostatic, incremental, and neomobil- 

istic) and models (synoptic, disjointed incremental, and planned change) to be ' 
served . 

4. Different types of decisions (planning, structuring, implementing, 

and re'cycling) require different types of evaluation designs, and a generalizable - 
and efficient evaluation model should be conceptualized accordingly. 

'5. While different evaluation designs vary in content, a sfngle set of 
generalizable steps (delineating, obtaining, and providing) can be followed. 

6. To answer questions posed by decision-makers , designs for evalua- 
tion studies should satisfy criteria of scientific adequacy (internal and external 

^^alid\ty, reliability, and objectivity), of practical utility (relevance, importance, 
scope, credibility, timeliness, and pervasiveness) and of prudential worth 
efficiency. 

7. Decision-making is comprised of four stages (awareness, design, 
choice, and action) that potentially require evaluative information; thus, the 
relationship between evaluattoiT and xiecision-making is symbiotic, 

8. Since decision-making requirements are subject to change, evaluation 
designs should be flexible and -capable of meeting changing requirements. 

Figure 10 is presented as an= overall model for the total evaluation program 
being proposed herein. It provides for systematic context evaluation and ad hoc 
■ input,. process, and product evaluations. This chart retains the basic relationships 

100 
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between activities, evaluation, and.decisions. The small loop (O) attached 
to each evaluation block denotes the general process of delineating, obtaining, 
and providing information that is inherent in any evaluation study . • 

The outer cycle represents a continuous, systematic context evaluation 
mechanism that provides both congruence and contingency context data. This 
mechanism delineates, obtains, and provides information to the planning body 
of a system to enable it to make decisions either to change the system or to 
continue with present procedures because they are serving important objectives 
effectively and efficiently, . ^ 

Figure 10, ^ 
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A 

The proposed model combines four evaluation and evaluation-related 
concepts, into a single generalizable model for evaluation. These are the three 
major steps in the evaluation process (delineating, obtaining, and providing), 
the three classes of change settings (homeostasis, imci-ementalism, and neo- 
mobilism), the four types of evaluation (context, input, process, and product), 
and the four types of decisions (planning, structuring, implementing, and re- 
cycling) . 

The fo(llowing features apply to this model: 

1. Adequate system evaluation requires the existence of a formal context 

evaluation mechanism; 

2. Leadership personnel in the context evaluation mechanism should 
have a continuing and direct relationship with those who formulate policy and 
planning decisions within the system. 

3/ Context evaluation should be based upon a data base appropriate for 

the relevant system. 

4 . '^\c context evaluation mechanism should provide both congruence 

and contingency data pertaining to the total system . 

5. Adequate resources should be at the disposal of the context evalua^ 
tion mechanism to enable the perform^© of ad hoc-input evaluation studies. 

6. Context evaluation is conducted in a general undifferentiated setting, 
while input, process and product" evaluations usually occur within specific, 

suboptimlzed settings. 

'7. In incr6mental and neomobilistic settings, structuring decisions 
should Include specifications and budgetary provisions for the conduct of process 
and product evaluation studies. jf^Q^ 
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8. Input, process, and product evaluation studies should be more 
comprehensive and highly structured for neomobilistic change than for incre- 
mental change. *. 

, 9. Input, process, and product evaluation to support honieo static change 
generally do not require formal evaluation studies by evaluation units. 

10. Process and product evaluation should be carried on simultaneously, 
,with process evaluation receiving heavier emphasis early in a trial and product 

evaluation receiving heavier emphasis as the trial progresses. 

11. Product evaluation assesses attainments of change projects within 
a system, and context evaluation assesses the impact of the change on the 
total system. 

12. All evaluation studies should follow the same general process of 
delineating, obtaining, and providing information. 

13. Generally speaking, evaluation and decision- making functions should 
be separated. To insure the objectivity of bqth ihe decisions and the evaluation, 
it is important that information be collected, organized, analyzed, and reported 
by persons trained in these functions, but they should not be responsible for 

the strengths and weaknesses of the program being evaluated. 

14. T6 maximize the effectiveness and efficiency of evaluation, evaluation 
itself should be evaluated. The criteria for this include internal validity, external 
validity, reliability, objectivity, relevance, importance, credibility, scope, 
pervasiveness, timeliness, and efficiency. 

In order to implement and maintain the functioning cff such a model it is 
necessary to develop evaluation teams similar to that proposed by Malcom Provus. 

' ' t 

The primary function of these team^^ato: 
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(a) play a role In the evaluation process, and 

(b) monitor the functioning of the evaluation model itself. 

This evaluation team will identify and collect information essential to 
decision-making and program improvement. The team is responsible for answering 
the following questions: 

1. Is the program defined ? 

2. Is the corrective action adequately defined? 

3. Is corrective action Installed? 

4. Is program installed?- 

5. Are the cjnabllng objectives being met? 

6. Are the terminal products delivered? 

These basic questions will be dealt with in stages. The following is a 
list of examples of questions to be raised and answered by the team, and a list 
of activities in which they will be engaged: 

\ 

' 1. Has the program been installed? ! 

2. Compare program definition with installation information for congruence 

3. Information about Installation obtained from field observations, 

4. Decide if program is congruent with standards. 

5. If program is not congruent, why has the program not been Installed? 

6. Model of program installation procedure. 

7. Description of actual installation procedure used. 

8. Decide where procedural breakdown exists. : 

9. What should be done to Install the program? 

10. Alternative Installation strategies of a general nature. 

11 Information about operational constraints on alternative strategies. 
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12. Select p'osslble specific strategies. 

13. What strategy Is best? ' ^ ' 

14. Value pr^loiities. of *he decision-maker. 

15. Estimates of the actual value consequences of each workable strategy. 

16. Selection of that strategy which optimizes values. 

17. " Is the program achieving its enabling objectives? 

18. Discrepancy information based on actual program performance of 

)• students. Yes, No. If not, why not? 

19 . Description oi breakdown points . 

20. What corrective alternatives appear possible under the model ?^<c»^ 

21. Create solution-set alternative's possible within the problem field. 

22. Detailed analysis of actual constraints In the problem field- * 

23. Choose from among' them the set which meets field requireirfents . 

24. What corrective alternative appears best? k i 

25. Information describing value consequences of alternatives. 
25. '-Is the corrective action adequately- defined? | 

27. Model of corrective action, deflnltlofi-adequacy criteria. 

28. Information descriptive of fisting corrective action definition. 

29. Determine if corrective action is adequate in terms of the model. 
If not, why not? 

30. Identify definition process actually used. 

31. Describe points at which the definition process has broken down. 

32. What corrective alternatives appear possible under the model? 

33. Create solution-set alternatives. 

11)5 



34. 'Detailed descrlptlon-anel^sls of problem field based oij problem- 

solving model. 

35. Choose from among them that set v/hlch satisfied field demands. • 

36. "What corrective alternative .appeafs best? 

37. Choose ^alterfiative .with best value-configuration fit. 

38. Is the corrective action installed?" 

39. vN^odel of corrective action (derived from definition of corrective action. 

40 . Information descriptive- of actua| field conditions . 

41. Determtne if congruence exists . 1 

A - 

■'. 42. If discrepancy, \yhy? ' (If not, why not?) 

43 . Ideritify breakdown, points (actual vs model discrepancies) . . 
Generally the Evaluation Team will consist of: 

\: Several nori-directive evaluation specialists skilled in small-group 
process lyork and ethnological techniques, each of whom has responsibility for 
project-evaluation mar^agement but all of whom may team up to facilitate group 

work. • 

2. One or more psychonietrists familiar with a wide range of group cogni- 
tive and affect- ve instruments and capable of rapidly designing ad hoc instruments. 

, 3. A research-design specialist capable of drawing carefully defined 
* ■' * 

samples, designing experiments, and directing the statistical analysis of data. 

4. One or more technical- writers. familiar with educational "language" 

and evaluation concepts. 

5. A data-processing unit with the capacity for data storage, retrieval, 

and statistical analysis as directed. 
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.6. Subject-specialist consultants. 

7. A status figure capable of communicating directly with' the superin- 
tendent of schools and all program directors. 

The following table depicts artivities of th'e Evaluation Team as they 
might relate to the activity of a program staff. 



Table I. 

Relation of Activities of the Evaluation and Program Staffs 
Evaluation-Staff Activity Program- Staff Activity 



Identify decision points in the entire 
evaluation process c Establish and 
maintain an apparatus whereby staff 
may formulate standards. 



Insure the adequacy of standards 
through the application of explicit 
criteria • 

Communicate statement of standards 
to staff. 

Identify Information needed to compare 
performance with standards. 
Design a method of obtaining program- 
performance information. 



Report standards vs. performance 
discrepancy. 

Identify decision points in the problem- 
solving process. 



Identify standards. 

Find ways in which to wprk with staff 
to reformulate standards/ if necessary. 

Find v;ay3 to resolve, differences in 
standards used by the program staff. 

Identify information available or attainable 
in order to compare performance with 
standards. * 

Provide information descriptive of program 
performance. 

Choose between action alternatives in 
regard to discrepancy. 

Identify kind of information needeid to 
identify cause of prograr» -performance 
deficiency. 



Locate information as to cause of pro- 
. gram-performance ^deficier^cy. 
Identify decision points in cncosing 
criteria to be used for selecting 
"possible'* and "best" corrective 
alternatives. "l^OT 
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t Explicate the criteria used to identify 
-cause of discrepancy. 
Identify available corrective^^lternatives, 
Identify information needed to generate 
alternatives. 

Locate and synthesize information 

as requested. Identify criteria underlying choice of 

best alternative . 

Choose "best" alternative for corrective 
action . 



In order to evaluate, the team will gather together certain data. The 
data are likely to be from several quite different sources, gathered in several 
quite'different ways. Whether tlie immediate purpose is description or Judgment, 
three bodies of information should oe tapped. In the evaluation report it will be 
helpful to distinguish between antecedent, transaction, and outcome data. 

The evaluation t(am will develop Data Matrices. Data will be recorded 
on a descriptive matrix and a judgment matrix. . The evaluation team will collect 
data related to the contingencies between antecedents, transactions and out- 
comes. These data will be recorded in terms of description and judgment. 
Descriptions will involve "intents" and "observations" and the congruency 
between then. Judgment will involve general standards of quality end specific 
Judgments. 




/ 
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CHAPTER V 



.PISTRICT REORGANIZATION FOR 
IMPLEMENTATION OF EVALUATION MODEL 



ERIC 



In order to implement the preceding evaluation^model it wals^ necessary to 
reorganize the administrative structure and functioning of the subject school 
district. In response to budget limitations as well as tc the need to gain avail- 
afble internal e^^pertise and involvement of the existing staff in the evaluation 
model there was a philosophical switch from centralized toward decentralized 
organizational decision-making. 

Prior to the introduction of this model to^,the administration of the City 

t ' 
School District of New Rochelle , the table of organization resembled that of 

most school districts of comparable size in the northeast region of the country. 

The Superintendent headed an administrative team of assistants, directors and 

supervisors whose centralized authority extended into all phases and divisions 

of the district's operation. A schematic of that organization looked like thisv 

Figure 11. 
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^-'^ . In the reorganization the central administration was stieatnlihed, as a 

first step,' dividing the major areas of the district's operation Into four parts • 
Figure 12 describes this organizational structure following the reorganization 
design. ^ 

\ Figure 12. ' 
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As a next step, workshops were held wi.h the adipinistrative staff in- 
cludUig the Superintendent of Schools. The entire staff was in agreement with the 
proposed evaluation model and with their involvement through taking on district- 
wide areas of responsibility in their areas of expertise. The following tables 
describe the new organizational structure for administr-atxve staff. Their specific 
e asted as they relate to decision-making, accountability and evaluation. 
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Figure 13. ^ 
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Figure 14. 
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Figure 16. 
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A review of the foregoing tables indicates the limits to which the district 
has gone to decentralize its operation and assign specific areas of function and 
responsibility to individual administrators. Notice, for instance, how in 
Figure }Z the entire instructional program of the district, in regular and special 
program areas, has been broken up into discrete parts with a single individual 
In charge of each. It should be stated that the new organization was designed, 
at least In part, to satisfy a need for greater accountability. It also provided 
the framework for making the evaluation model operational. 
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Within the prior organizational structure there did not exist any Evalua- 
tion Department. Most of what passed for evaluation or research was directed 

I 
\ 

by the Director of Pupil Personnel Services. This office basically directed a 
district-wide testing program, and prepared statistical reports. 

In order to coordinate the district-wide evaluation model and direct the 
newly developed decentralized organizational structure a Department of Evaluation 
was developed. Basically the following organizational structure was adopted 
-and approved by the Superintendent and Board of Education. 

Figure 17, 
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The Evaluation Model had to be implemented - as it had been designed - 
in the following context: No new professional positions were to be created for 
the purpose of Evaluation. The Evaluation Program would have to be conducted 
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by members of the existing professional staff assisted byVlerlcal personnel who 
were being made available. This was not conceived to be an imposition of 
additional work for existing staff; rather it was a reorganization of^iunction . 
which would thereafter undertake any formal evaluation. 

i 

The direction of the evaluation, however, would be assumed by the 
Director of Pupil Personnel Services. The other department would join in the 
effort,, principally with those of Instruction, Data Processing and Public Informa 
tion to carry on the various aspects of the Evaluation Model. 

Additional clerical staff would be added to iiandle the necessary record 
keeping, tabulation and reporting of data collected in ^nnectlon with this work 
and funds in the amount of $40,000 would be reallocated >\rithln thp district's 
operating budget to support those activities. 

Information would be collected by the Evaluation Department stating all 
the following persons (a job description of each member of the department is 
supplied)* 

U Director of Evaluation (also Director of Pupil Personnel Services 
and head of Evaluation Team) 

2 . Evaluator^ Stati stician 

3. Program Evaluation Supervisor (also Assistant Superintendent for 
Instruction) 

4* Research Psychologist 

\ 

s ^ 

5. Graduate Intern 

6. Public Information Specialist 

Their function as a department Is as follows: 

1^5^ ' 
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1. To descriBfe and evaluate the effectiveness of selected educational 
programs funded from local, state, and federal sources and adminis- 
tered by the school district. 

2. To provide information on the overall performance of schools in 
relation, to systemwide objectives. 

■ 3'. To design and manage the systematic collection, storage, retrieval, 
and dissemination of data. 

4. To maintain and disseminate information on exemplary and innovative 
practices in education. 

5. To prpvide consulting services in research, assessment, and evalua- 
tion for teachers, principals, project directors, and central office 
administrators. 

6. To promote and encourage the use df the assessment, research, and 
evaluation capabilities of the district. 

7. To provide at the direction of the director such information as is deemed 
necessary to conduct formative or summative evaluation of any program 
within the system . 

The Evaluation Team consists of the Director of Pupil Personnel Services, 
the Assistant Superintendent for Instruction, an Elementary Principal with some 
background in curriculum development, a secondary school Department Chairman 
with a strength in Research and Statistics, and a central administrator with a 
background in Special Education. 

The descriptions of the roles of these individuals as members of the 
Evaluation department are as follows: 
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JOB DESCRIPTIONS 

Position Title : Director of Evaluation 

Responsible to : Superintend eijt of Schools 

Primary Functions : \^ 

The director of Evaluation shall be responsible for all matters relating to 



evaluation in the New Rochelle Public Schools. He shall serve as t^ae leader of the 
evaluation department. 
Major Responsibilities: 

1. Promotes and encourages the use of evaluation in education programs 

2. Controls the operation of the evaluation department 

3. Coordinates activities of the evaluation department with ouier informa- 
tion gathering services used by the school system 

4 Provides training in evaluation to evaluation department staff members 
and the use of evaluation information' to other members of the school 
system staff 

5. Serves in other areas as directed by the Superintendent 

Illustration of Kev Duties : 

1. Facllitating-the use of evaluation- in educational programs 



a. 



Identifying deqlslon makers to be served and projecting 
decision situations to be served by evaluation 
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b. Inviting various personnel to participate actively In evaluation 
to the degree specified in policies for evaluation 

c. Establishing rapport with participants In programs to be 
evaluated 

d. Identifying and stating the purposes for evaluation In specified 
programs 

e. Preparing and distributing information about the operation and 
capabilities of the evaluation department 

f Working with other administrators to identify and disseminate 
the capabilities of ^Ipfatlon for serving the school personnel 
O and the community 
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g .Searching for new areas of the educational program for which 
evaluation would be useful and Identifying sources of financial 
. and human respurc^ fqV expanding the evaluation program. 

Controlling the operations of the evaluation department 

a. Assigning evaluation tasks to department personnel 

b. Reviewing designs, Instruments, and reports 

c. Reviewing staff and resource requirements and projecting those 
requirements Into the future In order to plan and budget for 
evaluation activities 

a. Working with administrators to write evaluation selections of 
proposals 

e. Approving evaluation activities contracted with other agencies 

f . Arranging for Internal evaluation of department activities and 
evaluation personnel 

Coordinating evaluation services 

a. Coordinating evaluation activities within the evaluation depart- 
ment and other departments 

b. Working cooperatively with other administrators responsible 

for Information gathering to insure compatibility of information , 

forms and data handling procedures 
Providing training 

a. Arranging and supervising training of evaluation department 
personnel 

b. Arranging for inservice training of pr(j)fessional staff members 
in conducting individual evaluation activities 

c. Providing inservice education programs and conducting workshops 
to enable decision makers to use evaluation findings as the 
basis for decision making 

d. Identifying further areas of training in evaluation needed by 
professional personnel (such as training in program evaluation 
and re\dew techniques for propos. . writers) , and arranging for 
such training programs. 

Serve in other areas asldJrSited by the Superintendent 
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.Position- Title; Supervisor of Research Services (E valuator- Statistician) 

Responsible to: -Director of Evaluation 

Primary Functions: 

1 . To administer the institutional research activities of the department 

2. To administer the information systems program for the department 

3. To provide relevant information to program developers to assist in 
the planning process 

4. To promote the use of research information 

Illustration of Kev Duties : 

1. To administer the institutional research activities of the department 

a. Supervise the long- and short-range planning activities for 
research services 

b. Supervise and manage the steps involved in planning and 
conducting a research study, i.e., problem identification, 
research design formulation, data collection, analysis 
methodology, and information reporting 



c. 



Supervise and evaluate staff assigned to the research unit 

d. Plan and administer'the fiscal resources for research services 

e. Cooperate with the department director and other unit super- 
visors to coordinate the work performed by the department 

f . Assist the Director in the formulation of policies and procedures 
governing the operation of research services 

2. To administer the information systems program for the department 

a. Supervise the development ^nd implementation of a data collec- 
tion, storage and retrieval system for the department 

• 

b. In support of the evaluation division, design and implement 

an information system which will report student progress toward 
interim and final project objectives 

c. Administer the needs assessment mechanism for the system, 
e.g., student surveys, dropout study, follow-up study, 
building profil^.9nd educational audits, etc. 
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To provide relevant information to program developers to assist in 
the planning process 

a. Supervise the maintenance and development of the innovative 
practices bank and department library 

b. Supervise the review of programs, method and strategies 
according to the criteria provided by the prpgram developers 

f 

c. Conduct reviews of the literature 

d. Serve as resource person on matters related to decision making 
and planning strategies, i.e., Delphi Technique, PERT, and 
advocate-team techniques 

To promote the use of research information 

a. Provide inservlce for department and other professional staff 
members related to the design, collection, analysis, and 
use of research information 

b. Screen proposed research activity requests from external 
agencies 

c. Serve as resource person for the district on matters pertaining 
to research design , ^ 

d. Maintain liaison with other educational research institutions 
and projects 
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Position Title: Supervisor of Program Evaluation (Assistant Superintendent 

\ for Instruction) 

Responsible to: Director of Evaluation (In Evaluation only) 

Primary Functions : 

The Supervisor of Program Evaluation shall be responsible for designing 

and coordinating the implementation of designated evaluation activities. He shall 

direct evaluation projects which fall into the purview of the program evaluation 

. division. 

Major Re'st3onslbllltles: 

1. Focusing on evaluation information to be provided 

2. Designing information collection, organization, and analysis procedure^ 

3. Reporting evaluation information 

4. Administrating designated evaluation services 
Illustration of Key Duties : 

1. Focusing on evaluation information to be provided 

a. Working with administrators and the Director of Evaluation to 
' make explicit assumptions of the project to be evaluated and 

to check the validity of those assumptions 

b. Restructuring objectives or Information to be gathered Into 
measurable form and outlining variables and criteria for 
measurement 

c. Identifying additional information to be gathered and the time 
by which that information must be provided to be useful 

d. Working with administrators to Identify audiences for evaluation 
information and specifying means for providing appropriate 
information to them at the time that information is needed 

e. Summarizing policy considerations which will guide the operation 
of evaluation activities 
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Designing information collection, organization, and analysis procedures 

a. Verifying sources of information to be gathered 

b. In cooperation with other evaluation personnel, developing 
specifications for evaluation Instruments and selecting or 
constructing appropriate instruments for data collection 

c. Defining procedures for a^dminiscration (including sampling • 
procedure's to be employed and scheduling of information 
collection) 

d. Supervise the coding and organizing of gathered information 

e. Selecting statistical or other analystical procedures for per- [ 
forming data analysis and designing a means for performing 

the analysis / 

/ 

f . Working cooperatively with other school personnel who provide 
data processing services as needed to complete evaluation 
activities 

Reporting evaluation information 

a. Determining audiences and estimated deadlines for evaluation 
reports 

b. Specifying means for providing appropriate information and « 
format for evaluation reports 

c. Scheduling reporting of information to professional and lay 
audiences 

d. Presenting evaluation information to professional and lay 
audiences 

Administrating designated program evaluation services 

a. Monitoring budgets and schedules of evaluation activities 

b. Insuring the smooth flow of evaluation informaiion to identified 
decision makers 



c. Maintaining smooth working relationships with persons providing 
and obtaining evaluation information and following accept^ 
school system protocol 

d. Supervise and evaluate staff as;^ned to unit 

too 
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P osition Title; • Supervisor of Assessment Serviceis (Research Psychologist) 

Responsible to; Director ojf Evaluation 

Primary Functions ; * * 

The Supervisor of Assessment shall be responsible for the development, 

« 

Implementation, and supervision of the district's norm-referenced and crlterlon- 
referenced testing program. \^ 
Major Responsibilities; ^ 

1. Supervise the district's norm-referenced and qpiterion-referenced 
^ testing program 

' " ) 

2. Identify, screen, and file available instruments 

3. Select and/or develop tests and other measurement instruments to be 
employed by the department ^ 

Illustration of Key Duties ; 

1. Supervise the district's norm-referenced and criterion-referenced 
testing program 

I 

a. In cooperation with the Director of Evaluation and other school 
personnel, design and schedule the assessment program for 
the schQol district 

b. Provide inservice training in the administration, utilization, 
and interpretation of testing instruments 

c. Provide testing instruments to meet specialized needs of pro- 
fessional personnel for individual or group diagnosis and 
assessment 

d. Provide for pilot testing of instruments before they are employed 
in ongoing projects 

e. Review the current literature on ins^umentation and keep the 
staff abreast of latest developments in the field 

, f . Supervise and evaluate staff assigned to unit 



117. 

Identify, screen, and file instruments . . 

a* Identify criteria for instrument selection and establish* pro- 
cei^ures for screening instruments 

Provide for the proctirement of appropriate quantities of tests 
and testing materials to meet thS* needs of the system 

c. Develop and maintain records .of the use, strengths, and 
weaknesses of available instruments 

d. Create a complete file of commercially supplied and locally 
constructed instruments and collect data on their applica- 
bility to departmental efforts 

Select and/oiMdeveiop tests and other measurement instruments to 
be employed by the department 



a. 



Select or create instruments needed for research and evalua- 
tion activities 



b. Write and/or supervise the wrillng of acceptable items for 
instruments \ 



c. 



Prepare and/or supervise the preparation of directions for 
administering and scoring all instruments 

/ 
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Position Title: Research Assistant (Graduate student) 

Responsible To : Director of Evaluation (or his designee) 

Primary Functions: * ^ ^ 

The research assistant shall be responsible fof assisting in the data 
collection, data processing, and reporting activities of the department 
Major Responsibilities: ^ 

1. Data Collection and instrument development 

2/ Data analysis and reporting 

3, Data processing functions ^ 

4. Other departmental activities 
Illustration of Key Duties : 

1, Data collection and instrument development 

a. Collecting data as specified in research and evaluation designs 

b. Provide assistance in selecting samples for" studies and surveys 

c* Provide assistance in implementing research and evaluation 
designs ^- 

d. Assist in the development and field testing of instruments, i.e. 
criterion-referenced tests and questionnaires 

1. Data analysis and reporting * 

[ a.^ Assist in planning research, evaluation or assessment project, 

, i.e., PERT, flow charting, task analysis 

\ 

b. Specify methods for collecting data 

c. Determine fornnat and code data in a form useful for analysis 
purposes 

d. Train paraprofessionals to gather and organize data 
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e* U$e analytical techniques and available hand and machine 
data processing devices to analyze information 

£• Prepare graphs, charts and tables to illustrate technical 
reports ^ 

g. Report writing 

\ 

Data processing functions . * 

a. Provide assistance in coordinating departmental activities 
with data processing personnel # 

V 

b. Provide a format for coding data 

c. . .Interpret computer printouts 

Assist in all phases of departmental activities 

a. Prepare reviev/s of literature 

b. Keep abreast of developments in tiie field of research, 
evaluation and assessment 
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Position Title: Public Information Specialist 

Responsible to: Director of Evaluation (or his designee) 

Primary Functions ; ' - 

The, Public Information Specialist is responsible for providing oral and 

written dissemination of departmental reports to persons both within and outside 

If 

the school system. 
Major Responsibilities: 

1. Disseminate information concerning departmental activities to outside 
groups through the use of written, oral, and visual communication 

2. Prepare reviews of litereture 

3. Undertake other communication projects, as needed ^ * 
Illustration of Key Duties : ^^^-^ — ► 

1. Preparing departmental reports x ^ 

a. DevWop and disseminate clear and o^cise departmental 
reports 

b. Assist other departmental personnel in writing and Siting 
their manuscripts 

c. Specify the audiences to*receive departmental reports based 
upon information provided byJtheMirector of evaluation 

d. Provide abstracts of departmental report? for presentation 
to specific groups 

e. Analyze the information requirements and reading levels of 
^ various audiences for evaluation reports 

f . Prepare findings and recommendations in a variety of interesting 
and understandable forms, such as: written reports, video 
tapes, wall charts, and overhead transparencies 

g. Describe the effect of different media on the message to be 
communicat-zd 

h. Obtain recipient's reactions to reports 
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2. Conduct, reviews of literature 

a. Deterlnlne specific areas of the topic to be covered 
h\ Identify sources of literature 

c. Synthesize and write the review 

d. Disseminate review to the appropriate audiences 

3. Disseminate Information about departmental activities 

a. Design and write a periodic departmental jiews letter 

b. Promote better communication and understanding between 
outside audiences and the department 

c. Develop and implement a system for continually informing 
syste^ personnel and the community about the work of the 
department 

d. Prepare an annual report on c/epartinental activities, which 
includes the goals, services, and overall activities of^the 
evaluation deparlineut , , 

4.. Undertake other departmental projects * . 

a. Prepare articles for outside publications 

b. jProvide technical communication assistance to other profes- 

sionals within the system 
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CHAPTER VI 
EVALUATION IN-SERVICE 

.District staff were Involved in Evaluation In-Service programs designed 
to acquaint them with the definition of evaluation and the rationale for an evalua- 
tion program in the school district* 

Participants in the Evaluation In-Service were^made aware of the evalua- 
tion types and the relationship of these types to decision making. 

Specific areas covered in the in-service included: 

1 . The New Rochelle Model 

2 . Focusing the evaluation 

^ . 3. Collection of information 

4 . Organization of information 

5. Analysis of information 

6. Reporting of inforni^ation 

7» Administration of the evaluation 

Participants were made aware of the interface relationship between 
evaluation and decision making and were exposed to evaluation and decision- 
maker responsibilities in the delineating, obtaining, and providing ihformation 
for context, input, process and product evaluations. 

Participants were shown the relationship between decision making and 
accountability. 

Also they were exposed to the methodology of evaluation and tasks for: 

1. Preparation of reports 

2 . Analysis of data 

123 . " 
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3. Organizing data 

4. Statement of evaluation policies 

5. Statement^ evaluation assumptions 

6. Specification of decisions 

Audio-Visual materials developed for these in-service programs are to 
be found listed in Appendix B and are available in" a supplement to this report. . 
Additional in-service materials used included programmed test questions, developed 
by staff, and are to be found in the following pages. 
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IN-SERVICE MATERIALS 

This section contains objective questions and keyed respo^nses to 
questions relating to New Rochelle's Evaluation Model. These assisted the 
workshop participants in determining whether they had achieved the objectives 
for this training. Each question appeared on a separate page in the training 
sessions and was followed on the subsequent page by a presentation and discus- 
sion of the correct and incorrect responses. In addition practical exercises were 
presented. In the interest of conserving space these appear continuously below 

e 

and not on separate pages as in the training sessions^ 
Question 1 . 

Which of the following is the most unique characteristic of the CIPP 

Model? 

a. It focuses on providing information for the major types of educational 
decisions . 

b. It considers the unmet needs of the school system. 

c. It evaluates outcomes in terms of their behaa/iorally stated objectives 

d. "r emphasized the role of the evaluator as decision maker. 

(After you have circled what you consider the correct response turn the pac,e to 
check your answer.) 
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Key for Question 1 



(Correct Response) 
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You should have circled "a" 

T I^A most unique characteristic of the CIPP Model is^that "it focuses on 
providing In formation for malor tvoes Oi ed ucational decisions," 

This is correct because CIPP's main departure from classical evaluation 
theory is to require that evaluation provide timely information for decision making. 

(Incorrect Responses) 
"b" is not the best response because it focuses on only one type of 
information that is gathered through use of the CIPP Model. 

"c" is incorrect because it denotes the main feature of the Tylerian 
. approach, which does not focus on decision making. 

"d" is incorrect because CIPP distinguishes sharply between the roles 
0|f evaluator and decision maker. 

Question 2 . 

Which of the following is not. a potential advantage of equating evalua- 
tion to measurement ? 

a. In the measurement approach, evaluation reports are based upon 

objective data. 

b. Data obtained under measurement approach to evaluation usually 
meet the assumptions required for interval and ordinal scales. 
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r. Under the measurement approach, most variables of Interest can be 
considered in the evaluation. 

d. The task of developing objective bases for judging program or pupil 

performance is quite feasible. 

I 

Key for Question 2 . 

(Correct Response) 



You should have. chosen "c" 

ft is not a main advantage of the mea surement approach that most 
variables of interest can be considered in the evaluation. 



This is so because the measurement approach directs educators to use 
liable tests that have been carefully validated. By following this advice the 
evaluator cannot consider many variable for which tests have not been developed. 



ava 



■ (Incorrect Responses) 
"a" is an advantage of equating evaluation to measurement because the 
measurement approach offers sound procedures for objectivity in administering 

and scoring tests . 

"b" is also an advantage because the measurement approach is based 
on methods that strive toward meeting the requirements of ordinal and interval 
' scales . 

••d" is also an advantage because objective tests have been used widely 
to provide information for judging both pupil and program performance. 

133 
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Question 3 . ^ 

Which of the following is a likely consequence of equating evaluation to 

experimental design? 

a. Projects being evaluated will be modified and refined frequently based 

on continual feedback from the evaluation. 

b. The evaluation will provide explicitly for the assessment of project 

goals . 

c. The evaluation will provide in depth c'ase study findings concerning 
the operation of a single project design. 

d". The evaluation will provide relatively inequivocal findings concerning 
the relative performance of competing project designs. 

Key to Question 3 . 

(Correct Response) 



You should have responded "d" 

PvppHmpntal design i s a direct' approach to dete rmine and compare the 
effects of alternative procedures . 



(Incorrect Responses) 

"a" is incorrect because experimental design requires that treatments be 
held constant during the period of the experiment and because experiments provide 
findings only .at the end of the study. |^ 

"b" is incorrect because experimental design does not provide for per- 
forming needs assessment or in any other way judging goals. 
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"c" is incorrect because experimental design focuses on more than one 
treatment and assesses product as opposed to process. > 

#» 

Question 4 . 

Which of the following best illustrates the difficulties associated with 
<»quating evaluation to professional judgment? 

a. In this approach evaluators are not restricted in the variables they 
may consider in assessing the merit of a program, and they may consider variables 
for which no valid measuring devices exist. 

b. This approach provides only a flimsy data base for the professional 
judgments that are rendered, and these are notoriously unreliable. 

c. This approach lacks independence since the evaluators who make the 
professional judgments usually are the persons who are in charge of the programs 
being evaluated . 

d. This approach is excessively expensive, in terms of both time and 

money . 



Key for Question 4 , 

(Correct Response) 



You should have circled •'b" 
The Professional judgment approach to evaluation prov ides only a flimsy \ 
data base for the professional judgments that are rendered and these are 



notoriously unreliable. 



This has been borne out in many studies of the interjudge reliability of 
project review panels and doctoral examination committees. 
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(Incorrect Responses) 
i" is Incorrect 'because not being restricted to variables for Which 
valid measures exist potentially enhances the validity of the study, since all- 
relevant variables may be considered in arriving at judgments. 

"c" is not correct because the professional judgment approach does 
utili - e outside, independent judges. 

"d" is not correct because the professional judgment approach actually 
is cheaper and less time consuming than most other evaluation api oaches. 

Question 5 . 

Which <S|f the following best characterizes the levels problem? 

a. Evaluators do not properly aggregats data gathered at a system's 
micro level so that these same data can be applied to assist decision making . 
at the macro level of the system. 

b. Evaluators do not determine what information is needed at each level 
of the system before designing their data gathering and analysis activities . 

c. Evaluators do net control their evaluation reports for appropriate 

I 

levels of readability. 

d. Evaluators do not properly disaggregate data gathered at a system's 
macro level so that these same data can be used to answer specific questions 
at micro levels of the system. 
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Key for Question 5 . 

(Correct Response) 



You should have responded "b" ' ' 

The levels pr oblem is that evaluators do not determin e what information 

^ — ^ n 

is needed at each level of the system before designing their data gathering and 



analysis activities . 



Since information requirements vary significantly across system levels, 
evaluations designeV^o serve one level^likely will not meet the needs of other 



levels . 



(Incorrect Responses)^ 

"a" is not correct because it assumes incorrectly that data gathered to 
serve the needs of the micro level of a system, is properly aggregated, will be 
sufficient to serve the needs of higher levels of the system. 

"c" is incorrect because it denotes a different problem from what has 
been^ermed. the levels program. 

"d" is incorrect for two reasons. It assumes incorrectly that data 
gathered to serve one level of questions can be disaggregated to serve more 
specific questions. It also assumes incorrectly that the information requirements 
of a macro level encompass thoso of lower levels of the system. 
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Question 6 . 

Which of the following best represents the CIPP conception of the evalua- 

tion process ? ' ^ 

a. Determine the operationally defined objectives, gather relevant 

outcome data, and compard" outcomes with objectives. 

b. Determine the questions to be answered, obtain relevant information, 
process and interpret the information, and provide feedback. 

c. Describe he antecedents , monitor the transactions , measure the 
outcomes, and interpret the full set of information. 

d. Focus the study, obtain appropriate information, aluate the informa- 
tion, and select the best action alternatives. 

Key for Question 6 . 

(Correct ResponseT 



You should have answered "b" 

The main step?; m the CiPP-defined evaluation process are delineating , 
obtaining, and providing information. 



These steps are best illustrated by the activities identified in "b". 



(Incorrect Responses) 
"a" IS not correct because it illustrates the process that is recommended 
in the Tylerian approach . 

"c" Is not correct because it illustrates not CIPP, but some of the steps 

! 

that are recommended by Stake in his Countenance Model. 
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*'d" is not correct because it emphasizes Scriven's point that eva^ators 
should decide what actions should be taken based on their evaluations. 



Question 7 . 

Using the CIPP criteria cited for judging the worth of an evaluation study, 
which^of the following statements best describes an adequate -evaluation study? 

a. It provides data which are unequivocal and which possess a high 
degree of generalizability . 

b. It is generaljzable to a specified set of conditions and a specified 

population of subjects. 

c. It provides a relevant rationale for action choices which is timely, 

def e ns ib le , a nd ef f ic ie nt . 

d. It provides information which is free from effects due to history, 
maturation of subjects, instrumentation, laboratory anrangements , and initial 
differences between comparison groups. 
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Key for Question 7 . 



(Correct Response) 



You should have answered "c" 

According to CIPP an adequate evaluation study should provide a rele- 
vant rationale for action choices which is timely, defensible, and efficient. 



This is the best response because it denotes the need for sound evalua- 
tion studies to provide information that is technically adequate, useful and cost 
effective. These are the three main standards that CIPP prescribes for sound 

evaluation designs and reports. 
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(Incorrect Responses) 
"a" is not correct because it is incomplete. It requires that evaluations 

* i 

be technically adequate but does n..t mention utility and cost/effectiveness . 

"b" is incorrect because it is also incomplete. It requires only that 
evaluations be externally valid. It does not mention internal validity, utility 
or cost/effectiveness. 

"."d!' is incorrect because it deals only with the internal validity portion 
of technical adequacy, and does not mention utility and cost/effectiveness. 

/ 

/ 

Question 8 . 

According to the PDK book, decisions ^hich specify procedure, personnel, 
facilities, budget, and time requirements are: 

a . planning decisions 

b. structuring decisions 

c . implementing decisions 

d. recycling decisions ^ 

Key for Question 3 . n 

(Correct Response) 



You should have answered "b" 

T^on^c^^n^ v.hich specif y nrncednres. -personnel, facilities, budget , 
md time requirements are structuring decisions. 



This is consir,tent with the CIPP position that structuring decisions are 
those that specify what procedural design should be implemented to achieve 
given objectives . 
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(Incorrect Responses) 
"a" is not correct because planning decisions specify not designs but , 
objectives. 

"c" is not correct because implementing decisions are those for carrying 
out chosen procedural designs. 

"d" is not correct because'recycling decisions are those based on the 
results of trying a given design, and then deciding whether, and if so how, to 
contiPU^3 using the design. 

OUfestion 9 . 

A decision which results in continuation of a project. beyond its initial 
funding period is called a: ' ^ 

a. planning decision 

recycling decision 

continuation decision 
d. implementing decision 

Key for Question 9 . 

(Correct Response) 

— 7 ' 

You should have answered "b" 

Decisions which result in continuation (or termination) of projects 
beyond their initial funding periods in CIPP terminology are called recycling 
decisions . , 
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(Incorrect Responses) 

"a" is not correct because planning decisions are those that spell out 
the initial objectives of special projects and not the decisions that determine 
whether to terminate, modify, or continue the project after its implementation. 

"c" is incorrect, a "continuation decision" is not one of the CIPP terms. 

"d" is not correct because implementing decisions do not determine 
whether a project will be phased out or repeated. Instead implementing deci- 
sions concern how a project design is, to be carried out. 
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ypuestion 10 . 

According to CIPP evaluation theory, what kind of decision is designed 
to answer any combination of the following questions: Should the project staff 
be retrained? Should new procedures be instituted? Should the project schedule 

be modified? 

a* structuring decision 
b* planning decision 

c. recycling decision 

d . implementing decision 



Key^for Question 10 * 



y 



;h^i 



(Correct Response) 



You should have answered "d" 
Decisions that involve the retraining of project staff, the institu ting 
of new project procedures . and the modification of project sche dules - 
according to the CIPP Model are Implementin g decisions. 
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(Incorrect Responses) 
"a" is. not correct because structuring decisions specify initial project 
designs, whereas implementing decisions concern how the designs should be 
carried out. 

"b" is incorrect because planning decisions spell out objectives and are 
not directly concerned with what procedures will be used to achieve the objectives 

"c" is incorrect because recycling decisions concern whether a project 
will be repeated and/or institutionalized, not how the project design is to be 
operationalized . 

Question 11 . 

Based on a study of student needs, a school board decides to assign its 
highest priority to improving school services for children with learning disabili- 
ties. This is a: 

a. planning decision 

b. structuring decision 

c . implementing decision 

d . recycling decision 

Key for Question II . 

(Correct Re^sponse) 



You should have answered "a" 

nori.ions that deter m ine 3ls , objectives, priorities, ends, etc. 

p{r& planning decisions. 

Ideally, planning decisions are based on context evaluations that 

reveal important needs. 
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(Incorrect Response^) 

••b" is not ^on-ect because structuring decisions determine not what 
objectives should be served but how given objectives should be achieved. 

"c" is incorrect because implementing decisions concern how to carry 
out a given project design to achieve given objectives. 

"d" is incorrect because recycling decisions concern not what objectives 
should be pursued but whether a special project already instituted to achieve 
given objectives should be cancelled or continued. 



\ 

\ 



Question 12 . » 

The board of education of a large city school district requests assistance 
in determining the causes of the high dropout rate in their district. Select the 
most appropriate type of evaluation . 

a. Context evaluation 

b. Input evaluation 

c. Process evaluation 

d. Product evaluation 

Key for Question 12 . 

(Correct Responses) 



You should have answered "a" 
T^ort^f.Hnr7 nnoH^ RrA diaanosinc; their ur.derlvina causes are rr.ain 



functions of context evaluation. 



Hence, context evaluation would be the appropriate way to determine 
the causes of a district's high dropout rate. ^ 
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(Incorrect Responses) 
"b" is Incorrect because input evaluation assesses alternative responses 
to problems once they have been determined through context evaluation. 

"c" is not correct because process evaluation assesses the implementa- 
tion of a given strategy (identified through input evaluation) for solving given 
problems (diagnosed through context evaluation) . 

"d"' is not correct because product evaluation assesses the results of a 
process that has been designed to solve certain problems. Hence, product 
evaluation in this case would assess the results of a special effort to reduce 
the dropout rate . 

Question 1.3 .- 

A school superintendent requests help in finding out whether a new 
language laboratory is being used in accordance with the specifications for its 
use. Which type of evaluation best responds to the superintendent's requests ? 

a. Context evaluation 

b. Input evaluation 

c. Process evaluation 

d. Product evaluation 
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Key for Question 13 . 

(Qorr^ct Response) 



You should have responded "c" 

Process evaluation assess the extent to which designs actually are 

/ 



being carried out, 

In this case process evaluation would assist the superintendent to 
ident'^IQ^ any discrepancies between his district's actual and planned use of the 
language laboratory. 



(Incorrect Responses) 
"a" is not correct because a context evaluation fundamentally is concerned 
with student needs and not one particular response, such as a language laboratory. 

"b" is Incorrect because input evaluation assesses alternative procedural 
designs for serving given objectives, input evaluations do not assess the ongoing 
operation of a chosen procedural design. 

. "d" is not correct because product evaluations monitor results not proce- 

dures. ' 
Question 14 . 

Given the following situation: A curriculum committee wishes to determine 
whether a special tutorial project is aiding as intended in the reduction of the 
schoors dropout rate. Select the most appropriate type of evaluation. 
^ a. Context evaluation 

b. Input evaluation 
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c. Process evaluation 

d. Product evaluation 

Key to Question 14 , 

(Correct Response) 



You should have responded "d" 

Product evaluations describe and judge proje ct outcomes > 
Thus, determining whether a tutorial project is aiding in i;«ducing a 
school's dropout problem calls for a product evaluation. 



(Incorrect Responses) 
"a" is incorrect because context evaluations focus on system needs, 
problems and opportunities as opposed to the attainments of special projects. 

"b" is not correct because -input evaluations assess procedural plans as 

opposed to project results. 

"c" is incorrect because process evaluations assess whether designs 
being implemented as opposed to what results are being achieved. 



are 



Question 15 . . < 

A school curriculum committee identifies and assesses the relative merits 
of several available curricula for high school physics. What type of evaluation 

is involved? 

a . Context evaluation ^ 

b. Input evaluation 



l-?t7 
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c. Process evaluation 
d* Product evaluation 

Key for Question 15. 

(Correct Response) 



You should have responded "b" 

In put evaluations identify and assess the relative m erits of competing 

strategies and designs . 

Thus, an input evaluation would be involved in identifying and assess- 
ing alternative curricula for high school physics. 



(Incorrect Responses) 

"a" is not correct because context evaluations assess system needs, 
problems, and opportunities as opposed to assessing alternative strategies and 
designs for responding to targeted needs, problems, and opportunities. 

"c" is incorrect because process evaluation describes and judges the 
implementation of a given design as opposed to assisting in its initial selection. 

"d" is incorrect because product evaluation does not assist in choosing 
a procedural Resign, but assesses its results once it has been chosen and imple- 
mented - ' 

Question 16 . 

The two uses of the CIPP Model are (1) for decision making and (2) for 
acco'intability. NAThichof the following statements illustrates its use for account - 
* ability? I'?t8 
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a. The model provides -information on the needs, problems, and oppor- 
tunities of a system from which tfoals and objectives can be derived. 

b. The model provides i record of the objectives chosen and the bases 
for their choice . 

c. The model provides information on whether to terminate, continue, 

or modify a program. 

d. The model provides for thS monitoring to project activities so the- 

program can be improved as it is implemented. 
Key for Question 16 . 

.(Correct Response) 



You should have responded "b" 

Accountability is the ability to describe and defend past decision s and 
actions . 

Hence, providing a record of what objectives were chosen, which were 
rejected and why is a us^e of context evaluation to serve accountability. 



(Incorrect Responses) 

"a" is incorrect because it denotes not an instance of accountability but 
a use of context evaluation to assist in choosing objectives . 

"c" is incorrect because it denotes a use of product evaluation to serve 
a recycling decision as opposed to serving an accountability need. 

"d" is not correct because it denotes a use of process evaluation to 
assist in implementing a design instead of to assist in retrospectively describing, 
and judging the completed process. 

^ 10 
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Exercise 1 » •. 

The CIPP conceptualization of evaluation identified four types of evaluation 
and four types of decisions. Bslow is a brief description of a hypothetical project. 
Analyze this example to identify what instances of context, input, process, and 
product evaluation are involved in supporting instances of planning, strXicturing , 

implementing , and recycling decisions . Record your answers on the following 

4. 

response sheets. 

r 

A school district obtained funds to upgrade the teaching of 
instrumental music for disadvantaged children. 

. It had been ascertained that poor children rarely received oppor- 
tunities to develop their musical interests and abilities. This led 
to a decision to improve the music education opportunities available 
to these children. Further assessment revealed that tne district was 
• especially weak in the area of instrumental music offerings, so the 
district officials decided to concentrate on improving their instru- 
mental music offerings for disadvantaged students. 

A decision making committee was formed and charged to develop 
a proposal for external funding of an instrumental music project. 
They identified (and employed external consultants to judge) several 
possible strategies for improving the district's instrumental music 
offerings for the disadvantaged. Finally they decided (and got funds 
for) a Jlan to buy musical instruments for usfe by poor children, to 
employ five new instrumental music teachers, and institute a volun- 
teer program for talented musicians in the community to assist in the 
program . ' 

In carrying through the program, it was decided that only four 
new music teachers would be hired, and that the additional money 
would be used to buy more instruments. This decision was served 
by evaluation that indicated that fqur new teacjjers wf Id handle the 
teaching but that insufficient money was available to buy all the 
needed insLiuments. 

At the end of the project, evaluation indicated that the project 
had effectively served the instrumental music needs of disadvantaged 
children. The district board therefore appropriated regular funds to 
institutionalize the project. 
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Key for Exercise 1. 

Responses to^ercise 1 should include the following: 

a. Context evaluation was involved in determining that poor children 
rarely received opportunities to develop their musical interests and abilities. 
More context evaluation was involved in discovering that the district was 

especially weak in the area of instrumental music offerings, 

/ 

b. Planning decisions were involved first in deciding to improve music 
education opportunities available to poor children and second in deciding to 
concentrate on improving the instrumental music offerings. 

> c. ] ;nput evaluation was involved' in identifying and assessing several 
possible strategies for improving the district's instrumental music offerings 
for the disadvantaged. 

d. Structuring decisions were those that determined that instruments 
would be purchased, five teachers would be hired /a Volunteer program would 
be instituted, and sufficient funds would be provided to carr/ out these activi- 
ties. 

e. Prdcess evaluation was involved in determining that four Instead of 
five new teachers could carry out the project and that more money than origin^ 
ally allocated was needed to purchase Instruments. 

f . Implementing decisions were Involved in deciding that only four new 

music teachers would be hired , and that the savings would be used to buy more 

ft 

instruments . 

y. Product evaluation indicated that the project had effectively served 
the instrumental musl^ needs of disadvantaged children. 
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h. & rpcvclino decision was invoJvfed in the board's decision to appro- 
priate regular funds for institutionalizing ,the proj|ct. 

4» 



Exercise 2 . 

Assume that you direct ap evaluation service agency which conducts^ 
evaluations based. on the'ciPP Model. Develop a brief evaluation proposal in*^ 
response to the followirig letter. 

^ Jack and Jaqueline Justice Family Foundation 
112 12th Street 
W^averly, .iQwa 

Dear Mr. Mc Judge: 

Presently, Justice Family Foundation is implementing a three- 
phased program to accict private or independent colleges in our 
region. The program primarily attempts to assist colleges, in the 
field of improving their enrollment and retention rate, the efficiency 
of the teaching-learning process, and the exploitation of an unfami- 
liar area to most colleges at this time deferred giving. It is assumed 
that the Foundation will designate several million dollars to partici- 
pant colleges tp apply some of the innovative programs to their own 
situations for the specific reasons of increasing college revenue, 
decreasing the rate of expenditure, increasing and maintaining the 
quality of the educational product. 

Given this brief summary of the Foundations' program, I would 
like to indicate tOvyou that we are deeply interested in developing ' . 
an evaluation prcaram that will tell us, rather precisely., how effec- 
tive our grants have been . If you feel that your center could *3e of 
some service , I would encourage you to explain your ideas and out- 
line a specific proposal to the Foundation as to how you might tackle 
this very challenging problem-opportunity. 



Ke y for Exerci se ?.: 

Responses to exercise 2 should present and develop tne following points: 
a. Tne approach we recommend serves both decision making and account 



ability. 




b. The approach called CIPP includes four kinds of evaluation. 

c. The CIPP model should be described. 

d. Illustrations should be provided concerning how the CIPP Model 
applies to the request for assistance. 

e. Specific suggestions' should be given concerning how your service 
agency would respond to the request for service . 



\ 
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CHAPTER VII 



IMPLEMENTATION OF NEW ROCHELLE EVALUATION MODEL , 

During April of 1974, school districts throughout New "/ork State were 
informed of amendments to the Regulations of the CommissioneP of Education as 
well as changes in financial apportionment provisions of Chapter 241. These 
changes were effective August 26, 1974. \ 

Chapter 241 of the Laws of 1974 of New York"State provides for aid to 
elementary and secondary education for 1974-75. This legislation, contains 
departures from previo> aidjegislation which has significant^ impact on the 
activities of local school districts. Among the most notable d6.partures included 
is one which provides for additional weighting of aid for pupils with spedial 
education needs. These are pupils who have scored on the most recent' approved 
achievement test two grade levels or more below the norm for the grade level in 
"^hich such pupils are enrolled, in reading or maru^matics, or, with rqspect to 
pupils in grades two and below, whose most recent acceptable readiness or 
other test scores predict a serious deficiency in reading or mathematics by the 
time such- pupils have entered grade three . 

The regulations of the Commissioner of Education indicated that each 
school district which receives an apportionmert for pupils with special educa- 
' tional needs shall continue and/or develop educationally advantageous programs 
These instructional piograrr.s should include: 

1. Specific pupil learning objectives to provide significant improvement 
of'pupil educational deficiencies; 
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2 . Activities and services which are clearly designed to achieve pupil 
learning objectives in an efficient manner; 

3. Expenditures which are directly related to such activities and services. 
Each school district that received an additional apportionment of state aid 

for pupils with special educational needs was required to prepare and submit to 
the Commissioner acceptable plans describing the use of such apportionment. 
Such district plan was required to have an evaluation including, but not limited 
to, the following: ' , * 

a. number and grade span of pupils served by planned activities; 

b. priority needs, and the method by which such needs were 
identified; 

c. program objectives; 

d. descriptions of program activities; and 

e. the method to evaluate the extent to which the objectives of 
the programs have been achieved. 

^-The> Superintendent of Schools and his Cabinet saw this as an opportunity 
to implement the New Rochelle Evaluation Model we had developed. We did so 
in the follov;ing sequence: 

Step 1 - Appoint District Evaluation Team to this program. 

Evaluation Team 

Name Title 

*Seymour Samuels Director of Pupil Personnel Scr/iccs 

Richard Olcott , A^st. Supt . , Instruction 

LaRuth Gray Dept. Chairman - English 

Dr. Laura Ilarkum District Evaluator - Research 

Joseph Isidori Coordinator - Language Development 

Ruth Geldon Administrator - Funded Programs 

♦Leader >^ r-^ 
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Step 2 - Inform District Evaluation Team of needed evaluation of programs 
and services for pupils with special needs. 

Step 3 - Outline Duties of District Evaluation Team 

a. Provide overall guidance for the evaluation including operations, 
direction, Information, materials, data processing, statistical 
analysis and personnel. 

b. Provide guidance and support to the Program Staff in their role 

of receiving and providing useful information for making decisions. 

c. Identify decision points in the entire evaluation process. 

d. Establish and maintain an apparatus whereby staff may formulate 
standards (what is expected). 

e. Insure the adequacy of standards. 

f . Communicate statement of standards to staff. 

g. Design a method of obtaining program -performance information. 

h. Report standards vs. discrepancy. 

i. Identify decision points in the problem solving process, 
j . Locate information as to cause of program deficiency. 

k. Identify decision points in choosing criteria to be used for 

selecting possible and best corrective alternatives. 
1. Locate and synt|?esize information as requested. 
Step 4 - District Evaluation Team in conjunction with Supsantendent and 
Assistant Superintendent for Instruction choose Program Staff. 
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Step 5 - Appoint Program Staff 



Name 

!• Quick 
J, Pozzi 
R. Byrnes* 
L. Lyman 
I. Konoshima 
H. Grossman- 
, Dr. Warshaw 
N. Philips 
R. Jacobs 

♦Leader 



Title 

Principal 

Principal 

Principal 

Principal 

Principal 

Dept. Chairman 

Psychologist 

Teacher 

Teacher 



District-Wide Service 

Math Elementary 
Language Arts Elementary 
District -Wide Service 
Alternative Programs 
Secondary Curriculum 
Math 

Special Programs ' 
Math 

Language Arts 



Step 6 - Outline Duties of Program Staff 
a • Identify standards. 

b. Find ways in which to work with staff to reformulate standards 
if necessary. 

c. Find ways to resolve differences. 

d . Identify information available or attainable in order to compare 
actual performance with what is expected (standard). 

e. Provide to Evaluation Staff information descriptive of program 
perf orma nee . 

f . Choose between alternatives when there are discrepancies 
between performance and standards. 

g . Follow through to implement alternatives . 

h. Maintain liaison with evaluation staff to receive and provide 
useful information necessary for decision making. 

Step 7 - A meeting was held with the Program Staff and the District 
Evaluation Team in order to determine the scope and areas of evaluation neces- 
sary. The information available indicated that the proposed evaluation should 

XO 4 



151. 

be both micro- and macro-analytic. Through other channels, information was 
either not available or not in a retrievable form to assist in decision-making 
regarding programs for pupils with special needs. 

Step 8 - The Evaluation Team and Program Staff predicted the decisions 
that would have to be made. The decisions to be .made were further delineated 
through proposed questions. 

a. Planning Decisions 

1) How well are we doing in our programs for pupils with 

special needs ? 

2) What problems exist ? 

3) V/hat unmet needs exist for these pupils ? 

4) What improvement-oriented objectives should be developed 
to meet identified needs ? 

5) What objectives will receive support of the school and 
'■ community ? 

6) VvThich objectives are most feasible to achieve? 

b. Structuring Decisions 

1) Can we modify existing programs for pupils with special 

needs ? 

2) What alternative programs could we use? 

3) What programs are feasible and likely to succeed ? 

4) How should they be organized and staffed ? 

5) What programs already exist with relevance for meeting 
previously established objectives? 

6) What programs sl]jpyii^ be selected ? 



c. Implement in g Decisions 

1) Is anything going wrong in programs for pupils with special 

needs? 

2) Are the programs on schedule? 

3) Does the staff need training? 

4) Are the facilities and materials ^ing used adequately and 
appropriately? 

5) What major Wocedural barriers need to be overcome in the 
operation o£ the present programs? 

d. Recycling Decisions ^ | 

1) How well will our programs for pupils with special needs 

work? 

2) Are our objectives being achieved? 

3) To what extent were' the needs of pupils with special needs 
met as a result of our efforts and programs ? 

4) Should we continue, modify, or drop a program? 

Step 9 - In order to answer these questions and judge decision alternatives, 
the four types of evaluation (CIPP) were outlined. Information requirements and 
activities were developed. 



Decision 



Planning 



Evaluation 



Context 



Information Requirements - Activities 



Analysis of existing objectives, scores, 
staff and community concerns. Discrepancy 
between system goals and system per- 
formance. Diagnoses of problems which 
account for discrepancy. Meetings, surveys, 
system analysis, scho6l profile review of 
literature, and consultants. 
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Decision 



Evaluation, 



Structuring 



Implementing 



Recycling 



Input 



Process 



Product 



liiformation Requirements - Activities 



Analysis of existing programs and resources. 
Identification of successful outside programs 
and materials. Information from literature 
on strengths and weaknesses of programs. 
Statements from experts. Comparative cost 
data. Evaluation and Program Staff judgment, 
and visitations to other programs. 

Discrepancy evaluation by Evaluation and 
Program Staff re: staff support, conflicts 
in organization, budgets,, schedules, 
strengths and weaknesses of materials, 
misunderstanding of purposes, training for 
personnel. Reports, observation and judg- 
ment of Evaluation and Program Staff. 

Performance in' relation to expected outcomes 
and objectives. Cost and benefit of program 
activities. Inference about the causal rela- 
tionship between program and results. 
Successes of students with special needs - 
Pre- Post-test Iowa Test of Basic Skills. 



This evaluation will serve both proactive support for decision making and 
retroactive support for accountability. 



Input 



Process 



Product 



Decision 
Making 


Objectives 


Solution 
strategy 

Procedural 
design 


Implementation 


Termination 
Continuat ion 
Modification 
Installation 


Accountability 


Record of 
objectives 
and bases for 
their choice 


Record of 
strategy and 
design and 
reasons for 
their choice 


Record of the 
actual process 


Record of 
attainments 
and recycling 
decisions 



IGO 
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The following matrix serves to summarize the evaluation of programs and 
activities for pupils with special needs. 



Data Requirements 
for Accountability 


Evaluation Types 


Context 


Input 


Process 


Product 


What objectives were chosen ? 
Why? 

Were they adopted 
Were they achieved ? 
What designs were chosen? 
Why? 

Were they implemented? 
' What were their effects ? 

f ' 


X 

' X 
X 


X 

« 

X 
X 


X . 
X 


X 
X 



3tep 10 _ Forms were developed to delineate and obtain information. 
These include the School Profile, testing program and others. Samples are in 
the appendix. 

Step 11 - The foregoing ten QO) steps serve as the process whereby the 
^ — 7 

New Rochelle Evaluation Model was implemented to delineate, obtain, and 
provide useful information for judging decisic . altornatives regard., j programs 
for pupils with special needs. It is expected that this evaluation plan is both 
formative and summative. The process will continue during the 1974-75 school 
year. 

The same procedures used to implement the New Rochelle Evaluation Model, 
in the evaluation of district-wide programs for students with special needs, was 
followed in the evaluation of two programs at the building level. 

Hillcrest Elementary School - Spring Valley 

The staff of the Hillcrest Elementary School was provided with in-service 
training in the philosophy and metnods of the New Rochelle Evaluation Model. 
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The traditional organizational structure of the school was redesigned to include 
an Evaluation Team and a Program Staff Team, as in Figure 18 (see page 156). 

The\;ei for evaluation originated with the teaching staff. The teachers 
voiced concern about needed decisions regarding the school's K-6 reading 
program. The teachers needed more information on (a) the difficulty of the mate- 
rials as they related to the abilities of the students; (b) the reasons for the 
decline of New York State reading scores; (c) the reasons for the discrepancy 
between math and reading scores; and (d) discrepancy between actual reading 
performance and expected performance. Requirements and objectives have been 
given to both the Evaluation and Program Staff. A calendar of activities has 
been developed f6r the completion of the evaluation during the 1974-75 school 
year. 

Pennington F.lementarv School - Mount Vernon 

The staff of the Pennington Elementary School was Vovided with in- 
seivice training m the philosophy and methods of the New Rochelle Evaluation 
Model. The traditional role of staff in the school was expanded to include staff 
that would serve as the Pennington School Evaluation Team and Program Staff . 

The need for evaluation originated with the grade teachers and the reme- 
dial reading teacher. Both had expressed dissatisfaction with the presently used 
Ginn 360 Primary Basal Reading Program. Decisions had to be made regarding 
■ the continuation, modification or replacement of this program. It was realized 
that there was not enough useful information to assist in appropriate decisions. 
Preliminary )discussion also indicated that little was known of why this program , 
was chosen and what needs it hoped to meet. Teachers indicated that they were 
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going to switch to the Keys to Reading Program prior to the in-service progra 



evaluation. During the in-service prograriv the staff indicated their desire for 
a complete evaluation, using the New Rochelle Model, in order to judge decision 



alternatives. 

0*. 

The Program Staff appointed included: two first grade teachers, school 
nurse, remedial reading teacher, and a parent volunteer. 

The Evaluation Team appointed included Assistant Superintendent for 
Instruction, Director of Reading, Title I Administrator, and the school psycholo- 
gist . ' — 

Requirements and objectives have been designed for both Program Staff 
and Evaluation Team A calendar of activities has been developed for the comple- 
tion of the evaluation dyring the 1974-75 school year. 



The New Rochelle Evaludlion Model has been implemented on a district- 
wide level in the City School District of New Rochelle, and on a building level 
in the Hilicrest Elementary School, Spring Valley, ^nd the Pennington Elementary 
School, Mount Vernon. The district -wide evaluation focused on all programs for 
students with special needs, while the building level evaluation focused on ^ 



reading programs. The formative and summative evaluations, will be completed 
during the 1974-75 school year. All three evaluations should provide data 
(information) in order to answer the tollowing questions. 

1. What objectives were chosen? Why? Were they adopted? 




/ 



SUMMARY 



/ 



Were they achieved ? 



0 
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2 . What designs were chosen ?' Why? 'Were they implemented ? 

What were their effects ? 
This information will assist in more effective educational decisions. 
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CHAPTER VIII 

« 

EVALUATION OF PRACTICUM " ^ 

The evaluation of this practicum was based on {he ability of significant 
school district staff to perceive whether or not the practicum participants were 
able to complete successfully the activities that were necessary to implement 
the New Rocheile Evaluation Model. ■ 

A behavioral questionnaire was developed and distributed to significant 
staff. The results of this questionnaire are on the following pages. Although'' 
there are some slight differences in perceptions of the degree of implementation 
there was consistency in the perceptions that the practicum participants 
completed the activities necessary to develop and implement our evaluation 
model . 
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V EVALUATION REPORT 

Implementation of a District-Wide Evaluation Model 



Please indicate the degree to which the New Rochelle Nova Practicum 
Participants carried out the tasks or activities they proposed. Your indications 
should be determined thru interface contact as well as your reading of their 
papers and participation in their training sessions. 



Degree of 
Implementation 



Activity 


Very 

Observable 


Moderately 
Observable 


Not 

Observable 


Determined need for Evaluation Model 


/ 

y 


\ 




Investigated Evaluation Models 


' ./ — 






Described Evaluation Models 


y 




^ ^ 


Critically examined the Models 




y 




Developed a v/orkable Model 




■y 




Developed Training Materials 




y 




Provid^^d Tn-sr^rvice Training 




y 




Developed New Organizational 
Structure— -v^ 


J 






Developed Ne^Role Descriptions 


/ 






Developed GuideyBook 








Implementcd-the Model ' 




-1 — 





I 




(Signed) / 



(Titl/ij 
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EVALUATION REPORT 
I? Implementation of a District-Wide Evaluation Model 

Please indicate the degree to which the New Rochelle Nova Practicum 
Participants carried out the tasks or activities they proposed. Your indications 
should be determined thru interface contact as well as your reading of their 
papers and participation in their training sessions. 



Degree of 
Implementation 



ERIC 



Activity 


Very 

Observable 


Moderately 
Observable 


Not 

Observable 


Determined need for Evaluation Model 








Investigated Evaluation "Models 








Described Evaluation Models 


y 






Critically examined the Models 








Developed a v/orkable Model 








Developed Training MateriaU-'^^ 








Provided In-service Tra^ng 








Developed New Or^nizational 
Structure 








Developed New Role Descriptions 








Developed Guide Book 


X 






Implemented the Model 


< 








( 






^' ' / (SiancdV 








fritio) 

■ J 





162. 

EVALUATION REPORT 
Implementation of a District-Wide Evaluation Model 

Please Indicate the degree to which the New Rochelle Nova Practlcum 
Participants carried out the tasks or activities they proposed. Your Indications 
should be determined thru interface contact as well as your reading of their 



papers and participation in their training sessions. 




Degree of 
Implementation 



Activity 


Very 

Observable 


Moderately 
Observable 


Not 

Observable 


Determined need for Evaluation Model 


X 






Investigated Evaluation Models 




Y 





Described Evaluation Models 


X 






Critically examined the Models 




X 




Developed a workable Model 


V 






Developed Training Materials 








Provided In-service Training 








Developed New Organizational 
Structure 


Y ^ 


i 

• 




Developed New Role Descriptions 


Y. 






Developed Guide Book 


Y 






Implemented the Model 


y 







(Signed) 
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EVALUATION REPORT 
Implementation of a District-Wide Evaluation I^^odel 

Please indicate the degree to which the New Rochelle Nova Practicum 
Participants carried out the tasks or activities they proposed. Your indications 
should be determined thru interface contact as well as your reading of their 
papers and participation in their training sessions. 



Degree of 
Implementation 



Ar*tlvitv 


Very 

Observable 


Moderately 
Observable 


Not 

Observable 


Determine^ need for Evaluation Model 








Investigated Evaluation Models 








Described Evaluation Models 








Critically examined the Models 








Developed a workable Model 








Developed Training Materials 








ProvidoJ In-Servica Training 








Developed New Organizational 
Structure 








Developed New Role Descriptions 








Developed Guide Book 








Implemented the Model 









(Signed) 



164. 

EVALUATION REPORT 
Implementation of a District-Wide Evaluation Model 

Please indicate the degree to which the New Rochelle Nova Practicum 
Participants carried out the tasks or activities they proposed. Your indications 
Should be determined thru interface contact as well as your reading of their 
papers and participation in their training sessions. 



Activity 


Degree of 
Implementation 


Very 

Observable 


Mouera teiy 
Observable 


Observable 


Determined rneed for Evaluation Model 








Investigated Evaluation Models 




X 




Described Evaluation Models 


> 






Critically examined the Models 




X 




Developed a workable Model 




T ' 




Developed Training Materials 








Provided In-service Training 








Developed New Organizational 
Structure 


X 






Developed New Role Descriptions 




-^3 

t 




Developed Guide Book 


X ^ 






Implemented the Model 


X 









(Signed)' 



(Ttrtio) 
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165. 

EVALUATION REPORT 
Implementation of a District-Wide Evaluation Model 

Please indicate the degree to which the New Rochelle Nova Practicum 
Participants carried out the tasks or activities they proposed. Your indications 
should be determined thru interface contact as well as your reading of their 
•papers and participation in their training sessions. 



Degree of 
Implementation 



Activity 


Very 

Observable 


Moderately 
Observable 


In Ol 

Observable 


Determined need for Evaluation Model 








Investigated Evaluation Models 








Described Evaluation Models 








Critically examined the Models 








Developed a workable Model 








Developed Training Materials 








Provided In-service Training 








Developed New Orvjanizational 
Structure 








Developed New Role Descriptions 








Developed Guide Book 








Implemented the Model 









(Signed) 
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EVALUATION REPORT 
Implementation of a District-Wide Evaluation Model 

Please indicate the degree to which the New Rochelle Nova Practicum 
Participants carried out the tasks or activities they proposed. Your indications 
should be determined thru interface contact as well as your reading of their 
papers and participation in their training sessions. 



Activity 


Degree of 
Implementation 


Very 

Observable 


Moderately 
Observable 


Not 

Observable 


Determined need for Evaluation Model 








Investigated Evaluation Models 








Described Evaluation Models 








Critically examined the Models 






1 — h 


Developed a workable Model 






/ 

— / — 


Developed Training Materials 








Provided In-service Training 




I 




Developed New Organizational 
Structure 


I .-^ 






Developed New Role Descriptions . 








Developed Guide Book 


y 






Implemented the Model 






n 




(Signed) 
(Titlqf ^ 
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EVALUATION REPORT 
Implementation of a District-Wide Evaluation Model 

Please Indibate the degree to which the New Rochelle Nova Practlcum 
Participants carried out the tasks or activities they proposed. Your Indications 
should be determined thru interface contact as well as your reading of their 
papers and participation in their training sessions. 



Degree of 
Implementation 



Activity 


Very 

Observable 


Observable 


Not 

Observable 




Determined need for Evaluation Model 








Investigated Evaluation Models 








Described Evaluation Models 








Critically examined the Models 








Developed a workable Model 








Developed Training Materials 








Provided In-service Training 








1 — . ■ 

Developed New Organizational 
Structure 








Developed New Role Descriptions 








Developed Guide Book 


y 






Implemented the Model 









(Signed) 




Title) 



168. 



EVALUATION REPORT 
Implementation of a District-Wide Evaluation Model 

Please indicate the degree to which the New Rochelle Nova Practicum 
Participants carried out the tasks or activities they proposed. Your indications 
should be determined thru interface contact as well as your reading of their 
papers and participation in their training sessions. 



Degree of 
Implementation 



Activity 


Very 

Observable 


[vi oaeraieiy 
Observable 


Observable 

[ 


Determined need for Evaluation Model 


X 






Investigated Evaluation Models 








Described Evaluation Models 








Critically examined the Models 








Developed a workable Model 


K 






Developed Training Materials 






— "V 


Provided In-service Training 


X 






Developed New Organizational 
Structure 


K 


• 




Developed New Role Descriptions 




c 




Developed Guide Book 


< 






Implemented the Model 


X 




' 8 




(Title) 
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Appendix B 

LIST or OVERHEADS IN ORDER OF PRESENTATION 

1. Rationale for Evaluation 

2. Methodology of Evaluation 

3. Developing Evaluation Designs 
4 . Definition of Evaluation 

5. The Relationship of Evaluation to Decision Making 

6. Decision Making as a Basis for Evaluation 

7. Decision Making Functions 

8. Symbiotic Relationship 

9. Evaluation 3cn.'iCGS Decision Making 

10. Evaluation awd Tecision Types 

11. Evaluation Types and Ccoratio^s 

12. A Dotailcd \vnlj-?t:cn Tv cdcl 

13. Iho CIP? Lval.. iiion Model 

14. Context Lvalaj.t:on 

15. Input i: valuation 

16. Procn^ss i/z-aluition 

17. Prod IK t Kv-il'da? ]-'\n 

18. i;v.iluatiOn 'ly;>-- ar:d Acco ji^.' ibi 1 ^ty 

!ntPif<u'C r/r^* !olr>':]^'n v;it!^ Decision !.':^):crs ^xwd/or Audienicos 

20. Int' ::Arr^, of r//i! : ^torf^ Wi^h P^c-ision !.Tikors and ^or A'jd!cn;c\n 

21. V.'< Pt' ■ i^yicv;:: for DolmcUirr; Iiiforr-u'ition 

22. rdanndnj Ciwn for Coll^^rMnp of ! ^-^sct i p* i vo DcUa 
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23. 


Tasks for Statement of Evaluation Assumptions 


24. 


Tasks for Statement of Evaluation Policies 


25. 


Tasks fo.r Specification of Decisions 

• 


26. 


Tasks for Analysis of Data 


27. 


Tasks for the Organization of Data 


28. 


Tasks for Preparation of Reports 


29'. 


Media Suggestions for Use in Reporting Evaluation Results 


30. 


Members of a Model Evaluation Unit 


3i. 


Organization of Evaluation Units 


32. 


Criteria for Assessing Evaluation • 



APPENDIX- C 
SAMPLE FORMS FOR DATA COLLECTION 



CITY SCHOOL DISTRICT 
City of Now Rochelle, Now York 
515 North Avenue • New Rochelle 10801 
Telephone — all schools and off.ces NE 2-9000 



Pupil Pe*'Sonnet Services 
Seymour Samuels, Director 



June 16, 1974 



TO: All Principalis 



FROM: Center for Educational Evaluation and Development 
RF: District Plcin 

In accordance with the new State f^id formOPla, the New York State 
Department of Education has mandated that all school districts submit a 
District Plan which will enable students with special educational needs to 
qualify for additional aid'monies. "Students classified as having special 
educational needs are. those pupils who have scored on their most recent 
'acceptable test at least two grade levels below the noun in reading or in 
mathematics, except in the case of pupils in grades 2 and below who shall 
be eligible for such supplemenlary programs on the basis of the^r most recent 
acceptable readiness or other test scores which predict t^eiious deficiency 
in reading or in mathematics by the time such pupils shall have reached 
grade 3 . " 

The purpose of this survey is to assist you and the district in 
determining priority pupil needs and program activities to^ meet thr.-so needs. 

According to the New York State P.E.P. results for the last two 
years, it is estimated that approximately the following percentage at each 
grade level in your school have special educational needs. 

School ■ Reading Math 

Wobster ~ 30% 37% 



Please outline the instructional strategies being used to nieet 
these; needs: 
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CITY SCHOOL DISTRICT OF NEW ROCHELLE 
CENTER FOR EDUCATIONAL EVALUATION AND DEVELOPMENT 

PROGRAM FOR STUDENTS WITH SPECIAL NEEDS 

This instrument* is designed to help assess the adequacy of a program. 

This is a checklist against which the administrator may*hold his* program 
for comparison. Following the checklist is a further list of questions . This 
section also includes a list of questions that help determine program compati- 
bility . 

Yes No 

1. Staff qualifications are sufficient for performing staff 

functions and duties. 

2. Staff duties are clearly related to staff functions. 

3. The administrative support' is sufficient for program 

operation, 

4. Media arc related to and sufficient for student 

activities, ^ 

5. Facilities are adequate for proqram operation. j . — ^ 

I 

6. The time allotted for program operation is sufficient 

to accomplish program goals. ^ ^ — ' 

7. Student activities are related to student goals. . r 

8. Staff act'vitics are related to student goals. — : — 

9. A process is defined ihat is sufficient to change 

each input into the output. ^ 

10. Com.munication activities within the program and 

between tr e Lio^^ram staff and others are sufficient 

to support L >eration, , 



^Adapted fiora Mik'om Fiovus' DEM ' 



180 



Questions : 

1. Do the objectives tell what the student does to signify success in the 
program ? 

2. ^*;^e the objectives clearly related to day-to-day activities of the program? 

' ^Z.- Are the entering behaviors consistent with the selection criteria of the 
participants? 

4, Are the entering behaviors linked to the program's objectives ? 

* 5. Are the media clearly related to activities ? Are they sufficient for the 
essential activities ? 

• 

'6. Are the activities related to the objectives ? Is there at least one activity 
for each objective ? 

7. Are time resources realistically related to the objectives r Is there sufficient 
^ time for each specified activity? 

8. Are staff qualifications and characteristics adequate for the functions 
defined ? If noi, are inservice activities spelled out to make them adequate? 

9. Are all functions necessary to serve the objectives included? 

10. Are^he functions for eat:h group clearly related to program objectives ? 

11 . Are the staff duties 'clearly related to staff functions ? 
'12. Are staff duties cloarly defiiiou ? 

13. Are there duties for each function? 

14. Arc intrastaff activities adequate for support of program operations ? for 
support of procaam objectives ? 

15. Are communication channels outside the program related to support needs? 
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CITY SCHOOL DISTRICT OT NEW ROCPIELLC 
CENTER FOR LDUCATIONAL EVALUATION AND DEVELOPMENT 



. ' TEACHLR PROGRESS REPORT 

Name of Teacher . School. 

Date 



Please answer each of the following questions to the best of your ability and qn 
the basis of your own personal experience. 

1. Ha§ a new practice aimed at coping with problems of students with special 
needs now been adopted m your school? 

Yes No 



2. , Are you aware of any noticeable benefit to you or your colleagues derived 
from the 'program adooted ? 



Yes No_ 



Yes No_ 



3. Are there any discernible benefit s;^.sUiclents ? 

4. Aie there any discernible benefits to admir.istrators ? Yes No^ 

Yes No_ 

If yes, please explain:. 



5. Has l\ been. necessary to change the program originally 
adopted ? 



6; Would you like to see an immediate change in the 
program now being used? 

If yes, what char/jC woukl you suggest ?nd why? 



Yes No_ 



7 . DO yc. a.liciral- that thq prca..:. will ro:n.in in eff^^ct 

as on:;i:ri!ly i:;',-r.d-'d for f.uo conVng i!-o-:'.h ..• >cs au_ 

If no, please explain:, 

8 . rlc:..':c .r: -.ko any other comnonts, yov. fool aro. relevont to the .success of 
tliC pio'jrain. 1.01. 



riTY SC'r'nOI. DISTRICT OF i\TW ROCHnLL.. 
CLNTER VOR EDUCATIONAL EVALUATION AND DEVELOPMENT 



List the in-servK-c staff development r.ctivities conducted m your school 
district in preparation for programs for siudents, with special needs., ..ease 
list each training topic, the number of personnel trained in each topic and 
the number of training hours devoted to each topic. 



TITLE OI' TOPIC 



Number of 
Personnel 
Trained 



Number of 
Training 
Hours 
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CITY SCHOOL DISTRICT OF NEW ROCHELLE ' 
CENTllR rOR EDUCATIONAL EVALUATION AND dCVELOPMENT 



Students v/ith Special Needs (Secondar>' School Principal) 



The District Evaluation Team is currently engaged in an evaluation project 
to determine the advantages and disadvantages of the implemen^ion of the 

Special Needs Program. This school year % of your stud^ntr> are students 

with special nfeeds . In order to discover the actual benefits a/id probJems 
associated with^our program, a wide variety of information othe^ than standardized 
test scores mu^ be collected. ^ ^ 

1 .^^Were' changes in type and/or quantity of instructional materials ^ ^ 

required to implemebt programs for these students? ' Yes _^No 

If your answer is yes , check these items affected: 

textbooks desks simulation goals 

' a udio-visual equipment work books trainincr packages 

research materials instructional guides 



library material: 



2. V/as rescheduling of course necessary ? Yes No 



How difficult v;as rescheduling with the addition of 

Special Programs compared to scheduUny courses prior to: L^ss^ 

y.^ ^ Same 

More 

Much mor^ 



3. If your staff participated in staff development preparatory prior to 
implementation of the program, how do you rate the value o/ the in- 
service sta!"; development ? (rate on an increasing seal*-, going from 
*'no vdluc^' CO "gif>y. value. 1 2 3 4 5 6 

4. What patterns of tirriO woie made available for staff, to raodify i^/jtrjc^jor.al 
patterns fbr students ^ 



/ 
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Principals 

I^st the in-service staff development activities conducted in yoiir building 
in preparation for the Special Needs Program fn the last 3 years. Please 
list each training topic; (the number of personnel trained, the number of 



Training hours, and the trainer.) 




t 


I 




Title of Topic 


Number of * 

Personnel 

Trained 


Number of 
Hours 


Trainer 








y 










i 
1 





/ 
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Interview Guide (Random sample) 
Counselor / 

Q. What has been the teacher's reaction to the Special Needs Program? » 

t 

/ 



How have pufjil^ reacted'? 



How have parents reacted ? 
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CITY SCHOOL DISTRICT OF NEW ROCHELLE 
CENTER FOR EDUCATIONAL EVALUATION AND DEVELOPMENT 

SPECIAL NEED:, ATTITUDE TOWARD LEARNING PROCESSES '(Elementary) 
CHECK: (• )BOY ( )GIRL' Grade 

We would like to know how you feel about how you learn in school. Blacken in 
the circle with a pencil to show how you feel. Fill in only one circle fop each 
question. YOUR TEACHER WILL NOT SEE THIS. Have fun! ^ 

r 

1 . We get enough time to help 

each other in class • NO SOMETIMES USUALLY YES 

0 0 O O 

2 . I have to spend too much time 

sitting at my desk NO SOMETIMES- USUALLY. YES 

O O , O^ O 

3. Everybody has to work on the ' • 

same thing at the same time.' •. NO SONIETIMES^^^ USUALLY YES 

O O ^'^ O O 

4. We get enough chances to choose the kinds 

of things we want to do in this class NO SOMETIMES USUALLY YES 

o q GO 

"5. We have to get permission from teachers ■ v \ 

to do anything around here NO SOMETIMES USUALLY YES- 

/ ■ • • O O \ O O 

6. We have enough chances to go outside 
the classroom and outside the school 
to learn things NO SOMETIMES USUALLY YES 

O O ; G O 

7 . If we ten the teacher what we would 

like to do, she tries to lot us do it NO SOMETIMES USUALLY YES 

O O G G 

8. Teachers do too much of the talking 

in class NO SOMETIMES USUALLY YES 

O G 0,0 

9. We have enough chances to walk 

around in the classroom NO SOMETIMES USUALLY YES 

O G GO 

U I . cont .... 



10. I have enough chances to sit together 

with my friends in this school NO 

O 

11. "We spend too much of our class time . 
reading the same books* over and 

over again ; NO 

O 

12 . . We- have too much homework . .[. NO 

I O 

13. I have enough chances to work with 

my friends in small groups.' NO 

O 

c 

14. I have enough chances to work on 

special things that I really like . . NO 

O 

15. I Wave enough chances to work as fast 
or/ as slow as I want to NO 

O 



SOMETIMES 
O 



SOMETIMES 
O 

SOMETIMES 
O 



SOMETIMES 
, O 



SOMETIMES 
O 



SOMETIMES 
O 



USUALLY YES. 



USUALLY YES 

O O 

USUALLY YES 

O O 



USUALLY YES 
O O 



USUALLY YES 
O O 



USUALLY YES 
O O 
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NEW ROCHELLE EVALUATION MODEL 



GUIDE 



"Evaluation is to Improve, not to Prove" 



Superintendent of SchooLs 
Robert R. Spillane, Ph.D. 



District Evaluation Team 
Gray, L.; Isidori, J,; 
Olcott, R.; Pozzi, J.; 
Samuels, S'. 



Center for Educational Evaluation and Davolopmcnt 
City School District of New Rochellc 
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GUIDE 

. to 

INITIATING EVALUATION THROUGW 
NEW ROCHELLE EVALUATION MODEL 



Introduction 



ERIC 



This guide has been prepared to provide the user with a systematic, 
^ step by step procedure for implementing desired evaluation . It p. .sents a brief 
description of what evaluation is as well- as the steps necessary in order to 
carry out cvaluati^'in the City School District of Nev/ Roc'helle. 

More detailed information can b^ found in the already developed New 
RochelJe Evaluation Model. ^ j 

. I 

What is Evalu ation ? ^ - ' . j ^ 

/ 

Evaluation is 'the process' of delineating, ottaining and providing useful 
information for making educational decisions. In line with this definition is 
the belief that evaluation is to improve rather than to prove. 

The process of evaluation depends upon the types of decisions one has 
to make. The decisions can be categorized into four types: 

a ^- PlQnnniq Decision s - These decisions specify the objectives to 

be achieved in an educational system. 

i 

2. Sjnjctur inp D ocisjons,- These decisions specify the moans to 
achieve the ends v/liich have been established as a result of 
planning decisions. 

3- il^Ul!air-yiUn2JlQ.ekl^^ Those decisions involve choices in 
carrying through tho action plon cstablishod by structuring deci- 
sion, 

20C 



4. Recyclin g Decisions - These decisions relate to choices in deter- 
mining whether to continue, ternrj^nate, evolve, or drastically modify 
the activity or program devoted to the solution of a system problem • 

jPecisi on Types ond Evaluat i on Typ^e?^ 

Each type of decision re<^uires a unique type of evaluation activities* 
Planning Dec ions ^ serviced by - Contexts-Evaluation 

Structuring Decisions - serviced by - Input Evaluation 
Implementing' -Decisions - servicbd by ^ Process Evaluation 
Recycling Decisions - serviced by - Product Evaluation 

J Context Eva l uati on - This type of evaluation is systematic and 

macroapdlytic. Its purpose is to provide a rationale for tho^ 
determination of objectives. It defines tfie environment, describe 
the dcro'rod and actual conditions pertaining to the environment, 
identifies unmet needs, and diagnoses the problems that prevent 
needs from being met. 
2. Inpu t Ev alu ation - This type of evaluation is essentially ad hoc 
and microanalytic. Its purpose is to provide information for 
determining how to utilise resources to meet program goals. It 
identifies and assesses relevant capabilities of the responsible 
individuals or agencies, stratcnios for achieving program goals, 
and dc5;igns for implementing a selected strategy. 
3- ii£iri!?ii:_LY/lU^^U2^^ *" Tlus type of evaluation provide^v periodic 
feedback to persons responsible for Implementing plan.s and pro- 
cedures. It has throe? objeclivos; (1) to detect or preciict defects 



in the procedural design or its implementation during the imple- 
mentation stages; (2) to provide information for programmed 
decisions, and (3) to maintain a record of the procedure as it 



occurs. 



4. Product Evaluation - This type of evaluation measures and inter- • 
pre^s Qttai^nments at the end of a program cycle. It assesses the 
extent, 'to which ends are being attained with respect to the change 
efforts. 

CIPP is an acronym formed from the first letters of the four oasic kinds 
1 

of evaluation: Context, Input, Process, Product, developed by Daniel Stufflebeam 
and incorporated as the basic framework for the New Rochellc Evaluation Model. 



/ 
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Evaluation is 



The Process of Delineating,. Obtaining and Providing Useful 



INFORMATION for making EDUCATIONAL DECISIONS 



ContexL \ . . . Planning 



Analysis of existing objectives 
of programs, scores, staff and 
community concerns and 
chargicteristic$ ; etc. 



ill < 



How ^V6ll are we doing in our' 
programs? WTiat problems exist? 
What changes in purposes should 
be made? 



Input 



Analysis of existing programs 
dnd r.esources, identification of 
successfuloout^ide programs and 
materials, information from 
literature cn strengths and 
weaknesses of programs, etc. 



Structuring 



*Can we modify existing programs? 
What alternative programs could 
we use?^ What programs are 
feasible and likely to succeed? 
How should t^iey be organized 
and staffed? What program would 
be selegted? 



Process ImMementinq 



. Imhl' 



r 



Staff s-upport, conflicts in 
organization, tight budgets and 
schedules, strengths and 

* weaknesses of materials, 

• misunderstanding, of purposes, 
lack of trained personnel. 



Is ai^ything going wrong in the' 
program ? Staff? Schedule ? 



Product . . . ' Recycling 



Performance in relation to 
expected* outcomes and 
ol^cctives, costs and benefits 
owcfifferent program activities, 
etc. 



How well did our prograi^ work? 
Should we continue, modify, 
or drop the program? 
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The Nov/ RochcllG Evaluation Model will provide both proactive support 
for decision making and retroactive support for accountabiMy . • 





Context 


Input 


Process 


Product 


Decision 
Making 


^ • ■ 1 

Objectives 


Solution 
strategy 

Procedu*al 
design ^ 


impierneriici cion 


i 

Continuatiori | 
Modification i 
or Installation ! 


^ j Record of 

1 objectives and 
Accountability, bc^ses for 

j their choice 

1 


Record of 
chosen strategy 
and design and 
reasons for 
their choice 


Record of 
the actual 
process 


Record of ♦ 
attainments 
and recycling 
decisions 



ThQ following matrix, developed by Daniel Stufflebeam, serves to summarize 



tho evaluation modeh 



Data Requirements 
for Accountability 



\ 

What objectives were chosen? 
Why? 

Were they cicioptod ? 
Were they achieved ? 

WheU dcr:igr5S v/orc chosen? 
Why? 

* Woio th.'-y iir.p]Gment{^d5 
What vv-^ro ihnir offsets? 




Evaluation Types 



Input 



X 



X 



Process 



Product 



X 



X 
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THE ROLE or ADMINISTRATORS AND TEACHERS 

The rola of the drj.'sicn- maker who gathers and v/elghs information prior 
» 

to making a choice and taJding action applies to all members of the administrativo 
and teaching stt.ff sinc^^ both function as decision-m.akors. Principals, teachers, 
and other designated professional personnel, individually or in cooperation Vviih 
others, have four broad obligations in the process of evaluation^ These obliga- 
tions deal with focusing on evaluation information tu hfe provided, providing 
information to mombersjpf the evaluation staff, handling evaluation information 
given to them, and using evaluation information provided lo them. 

The steps necessary to initiate and oomplote evaluation include: 

1. Staff, through the building piincipal, shall define a program or 

programs that require decisions. 
2*. Staff, through the building principal, vvill define the necessary 
decisions for which information will be needed, 

3. The piincipai will identify representative staff that are involved 
in- the program which is in need of evaluation. . 

4. This appointed staff will be cohsidered and function as the Program 

* 

Staff Team.. Activities of Program Staff Team may include: 
n. Identify standards 

b, Find ways in v/hich to worK with staff to reformulate 
standards if iiecesjary 

c, Tind wavs lo resolve difference 

d, Idenfiiy inforiTiation avoilal^le or attainable In order to 

t . 

coinF>"ire actual performcince with what ir c:<pected (standard) 
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e. Provide to Evaluation Staff information descriptive of 
program performance 

f. Choose between alternatives whenCthere are discrepancies 
: between performance and standard 

y. Follow through to implement alternatives 
\^ h. Maintain liaison with evaluation staff to receive and 

provide useful information necessary for decision 'making 
5. It is not expected that the program staff team can carry out " 
these activities without support of the district Evaluation Staff 
Team. 

o. The building principal secures the services of ih^istrict 
Evaluation Staff Team by filling-out the Evaluation Services 
Request Form. These forms are in the office of the building* 
principal. A copy of this form is attached to this guide. 

7. Any staff member may rcauest such evaluation services; however, 
reques^ts must be submilt^^d througii the building principal. 

8. The request for evaluation cen^iccs should be submitted to the 
Director cf Pupil Personnel Services - Center for Education^) 
Evaluation and Dcvcl^nient. ^ j 

9. The district Evaluation Staff Team v/ill proviac the overall 

. guidance for the evaluation including operations, direction, 

information, materials, data processing, statistical analysis, 

\ 

and pcrsonncj . 

'10. The district Evaluation Staff Team wiJl provide the guidance and 

X . . . 

support to the Program Staff in their ro^o of receiving and 
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providing useful information for making decisions. 
11. In addition, the following activities may also be performed by 
the Evaluation Staff Team: 

a. Identify decision points in the entire evaluation process 

b. Establish and maintain an apparatus whereby staff may 

i 

"V formulate standards (what is expected) 

c. Insure 'the adequacy of standards 

d. Communicate statement of standards to staff 

e. Identify information needed to compare pejrformance with 
standards , ^ 

f . Design a method of obtaining program-performance ■ 
information 

g. Report standards vs. performance discrepancy 

h. Identify decision points in thi problem solving process 

i. Locate information as to cause of program-performance 

deficiency 

j. Identify decision points in chosing^riteria J:o be used for 
selecting possible and best corrective alternatives 



k. Locate and synthesize infprmation as requesteqi. 



DEFINITION 

Educational cvoluatibn is the pioco^s c3f delineating, obtaining, and 
providing useful infonnalion for judging decision alternatives. 

"Evr.luntion is to improve, not to prove." 
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Appendix A 



Center for Educational Evaluation and Development 
Office of Pupil Personnel Services 



EVAyjATLQN.jSE.RiaCJES 
REQUEST FORM 



Name 



School^ 
Date 



Position 



Activit/Title_ 



/ 



Brief Descri ption of Activity (Program) : 



7 



Evaluat i on Services Desired - S p ecific Infonnation Needed : 



0/ 



\ 
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(Principal) 
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NEW ROCHELLE DISTRICT TESTING SCHEDULE 



/ 



City School District of New Rochclle 
Department of Pupil Personnel Services 
Office .of Psychological Services 



/ > 

Grade 


Administration 
Date 


Tyne of Test 


Name of Test 


Kg 


\Sept. 23-2'7 


Basic Concepts 


Boehm Test of Basic Concepts 


1 


\ — — — ' 

\ 

Sept. 23-27 ; Reading Readiness 


Murphy Durrell Reading Readiness 


2 


Drt IS- 18 ' Achievement 

} 

j 


Iowa Test of Basic Skills, 
Primary Form 5, Level 7 


3 


Sep.30-Oct.4 i Achievement 
Oct. 15-18 , Achievement 

i 
f 


N.Y. Stale Reading and Arithmetic 
Iowa Tesl of Basic Skills, 
Form 5, Level 8 


4 


Oct. 15-18 ! Achievement 

i 

i 

Nov. 11-15 Ability 


Iowa Test of Basic Skills, 
Form 5, Level 9 
Lorge-Thorndike 


5 


Oct. 15-18 , Achievement 


Iowa Test of Basic Skills, 
Form 5, level 10 


6 / 


Sep . 30-Oct . 4 Achievement 
Oct. 15-18 Achievement 

Nov. 11-15 Ability 

! 


N.Y. State Reading and Arithmetic 
Iowa Test of Basic Skills, 
Form 5, Level 11 ^ 
Lorg^nThorndike ^ 


7 


i Oct. 15-18 Achievement 

! ■ ■. _ 


Iowa Test of Basic Skills, 
Form 5, Level 12 


8 


Oct. 15-18 ; Achievement 
Doc. 3-6 Aptitude 


tov\)a Test of Basic Skills, 
Form 5, Level 13 
Differential Aptitude Test 


9 


SGp.30-Oct.4 Achicvomont j N.Y. State Reading and Arithmetic 
Oct. 15-18 Achieveniont j Iowa Test of Basic Skills, 
' i Form 5, Level 14 



OUT or SYSTEM TESTING 



OUT or SYSTEM TESTING 
OUT or SYSTEM TESTING 



/ 
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